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Preface 



The automotive industry with more than 2 million employees is not only one 
of the main pillars of European economy it is also of all economic sectors the 
largest R&D investor in Europe, realising 20% of the R&D expenditures of the 
entire manufacturing industry Through this it constitutes a major driver for 
the development and diffusion of new technologies and innovations through- 
out the economy 

Looking back 10 years when the International Forum on Advanced 
Microsystems for Automotive Application (AMAA) started, enormous progress 
has been made in reducing casualties, emissions and in increasing comfort and 
performance. Microsystems in many cases provided the key functions for this 
progress. Although the issues the event concentrated on didn't change signif- 
icantly (safety, powertrain, comfort, etc.), considerable shifts of technological 
paradigms and approaches can be stated. 

This year the AMAA will celebrate its 10 th anniversary, recapitulating objec- 
tives achieved and looking forward to new challenges for the automotive 
industry deriving from increasingly complex miniaturised systems driven by 
socio-economic requirements. 

The future of microsystems will consist of integrated smart systems which are 
able to diagnose a situation, to describe and to qualify it. They will be able to 
identify and mutually address each other. They will be predictive and there- 
fore they will be able to decide and help to decide. Smart systems will enable 
the automobile to interact with the environment, they will perform multiple 
tasks and assist a variety of activities. Smart systems will be highly reliable, 
often networked and energy autonomous. 

There is a coincidence of the AMAA objectives and those of EPoSS, the 
European Technology Platform on Smart Systems Integration, contributing 
intensively to the development of automotive-specific smart systems. You will 
find a series of the EPoSS items in the programme of the 10 th AMAA, which 
continues to be a unique exchange forum for companies in the automotive 
value chain. 

The publication in hand also reflects these issues. It is a cut-out of new tech- 
nological priorities in the area of microsystems-based smart devices and opens 
up a mid-term perspective of future smart systems applications in automo- 
biles. 



I would like to express my sincere thanks to the authors for their valuable con- 
tributions to this publication. I would like to thank the members of the AM A A 
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team at VDI/VDE-IT for their assistance with the conference, to Mr. Michael 
Strietzel for preparing the book for publication, and especially to Dr. Jurgen 
Valldorf, project manager and chairman of the AMAA. 

Berlin, April 2006 

Wolfgang Gessner 



Public Financers 



Berlin Senate for Economics and Technology 

European Commision 

Ministry for Economics Brandenburg 



Supporting Organisations 

Investitionsbank Berlin (IBB) 
mstnews 

ZVEI - Zentralverband Elektrotechnik- und Elektronikindustrie e.V. 
Hanser automotive electronic systems 
Micronews - The Yole Developpement Newsletter 
enablingMNT 



Co-Organisators 

European Council for Automotive R&D (EUCAR) 
European Association of Automotive Suppliers (CLEPA) 



Advanced driver assistance systems in Europe (ADASE) 



Honorary Commitee 



Domenico Bordone 


President and CEO 
Magneti Marelli S.P.A., Italy 


Gunter Hertel 


Vice President Research and Technology 
DaimlerChrysler AG, Germany 


Remi Kaiser 


Director Technology and Quality 
Delphi Automotive Systems Europe, France 


Nevio di Giusto 


President and CEO 
FIAT, Italy 


Karl-Thomas Neumann 


CEO, Member of the Executive Board 
Continental Automotive Systems, Germany 



Steering Commitee 



Dr. Giancarlo Alessandretti 
Alexander Bodensohn 

Serge Boverie 
Geoff Callow 

Bernhard Fuchsbauer 
Kay Furstenberg 
Wolfgang Gessner 
Roger Grace 

Henrik Jakobsen 
Horst Kornemann 

Hannu Laatikainen 
Dr. Torsten Mehlhorn 
Dr. Roland Miiller-Fiedler 
Paul Mulvanny 
Dr. Andy Noble 

Ulf Palmquist 
Gloria Pellischek 
David B. Rich 

Dr. Detlef E. Ricken 

Jean-Paul Rouet 
Christian Rousseau 
Patric Salomon 
Ernst Schmidt 
John P. Schuster 
Solzbacher, Florian 
Bob Sulouff 
Berthold Ulmer 
Egon Vetter 

Hans-Christian von der Wense 



Centro Ricerche FIAT, Orbassano, Italy 
DaimlerChrysler AG, 

Frankfurt am Main, Germany 
Siemens VDO Automotive, Toulouse, France 
Technical & Engineering Consulting, 
London, UK 

Audi AG, Ingolstadt, Germany 
IBEO GmbH, Hamburg, Germany 
VDWDE-IT, Berlin, Germany 
Roger Grace Associates, 

San Francisco, USA 

SensoNor A.S., Horten, Norway 

Continental Automotive Systems, 

Frankfurt am Main, Germany 

VTI Technologies Oy, Vantaa, Finland 

Investitionsbank Berlin, Berlin, Germany 

Robert Bosch GmbH, Stuttgart, Germany 

QinetiQ Ltd., Farnborough, UK 

Ricardo Consulting Engineers Ltd., 

Shoreham-by-Sea, UK 

EUCAR, Brussels, Belgium 

Clepa, Brussels, Belgium 

Delphi Delco Electronics Systems, 

Kokomo, USA 

Delphi Delco Electronics Europe GmbH, 
Riisselsheim, Germany 
Johnson Controls, Pontoise, France 
Renault S.A., Guyancourt, France 
4M2C, Berlin, Germany 
BMW AG, Munich, Germany 
Motorola Inc., Northbrook Illinois, USA 
University of Utah, Salt Lake City, USA 
Analog Devices Inc., Cambridge, USA 
DaimlerChrysler AG, Brussels, Belgium 
Ceramet Technologies, 

Melbourne, Australia 

Freescale GmbH, Munchen, Germany 



Conference chair: 



Dr. Jurgen Valldorf 



VDWDE-IT, Berlin, Germany 



Table of Contents 



Market 

Prospects for MST Sensors in Automotive Applications 

R. Dixon, J. Bouchaud, WTC - Wicht Technologie Consulting 

Future Architecture for Inertial Sensors in Cars 

J.C. Eloy, M. Potin, Dr. E. Mounier, Dr. P. Roussel, Yole Developpement 



Safety 

Three-Dimensional CMOS Image Sensor for Pedestrian 
Protection and Collision Mitigation 

P. Mengel, L. Listl, B. Konig, C. Toepfer, M. Pellkofer, Siemens AG 
W. Brockherde, B. Hosticka, O. Elkhalili, O. Schrey, W. Ulfig, Fraunhofer IMS 

EVENT-ONLINE - A Service Concept for large scale Events 
based on FCD Technology 

R. Willenbrock, F. Steinert, gedas deutschland GmbH 
W. Schonewolf, Fraunhofer IPK 

K. Graze, Signalbau Huber GmbH 

Advanced Pressure Sensors with high Flexibility for 
Side Crash Detection 

M. Brauer, Dr. K. Krupka, Infineon Technologies AG 

Detection of Road Users in Fused Sensor Data Streams for 
Collision Mitigation 

L. Walchshausl, R. Lindl, K. Vogel, BMW Group Research and Technology 
T. Tatschke, FORWISS Institute for Software Systems, University of Passau 

Dynamic Pass Prediction - A New Driver Assistance System for 
Superior and Safe Overtaking 

J. Loewenau, K. Gresser, D. Wisselmann, BMW Group Research and 
Technology 

W. Richter, BMW Group Development 
D. Rabel, S. Durekovic, NAVTEQ 



79 



Requirement Engineering for Active Safety Pedestrian Protection 
Systems based on Accident Research 

R. Framing, V. Schindler, TU Berlin 
M. Kuhn, VTIV 

Biologically Inspired Multi-Sensor Fusion for Adaptive Camera 
Stabilization in Driver-Assistance Systems 107 

W. Gunthner, S. Glasauer, Ph. Wagner, H. Ulbrich, TU Munich 

Low Speed Collision Avoidance System 1 23 

J. Kibbel, H. Salow, M. Dittmer, IBEO Automobile Sensor GmbH 

Laserscanner for Multiple Applications in Passenger 

Cars and Trucks 1 29 

R. Schulz, K. Furstenberg, IBEO Automobile Sensor GmbH 

A new Approach for Obstacle Detection Based on 

Dynamic Vehicle Behaviour 143 

M. StraBberger, BMW Group Forschung und Technik 

R. Lasowski, Softlab GmbH 

Far Infrared Detection Algorithms for Vulnerable 

Road Users Protection 1 55 

Y. Le Guilloux, R. Moreira, S. Khaskelman, J. Lonnoy, SAGEM 

iBolt Technology - A Weight Sensing System for 

Advanced Passenger Safety 171 

K. Kasten, A. Stratmann, M. Munz, K. Dirscherl, S. Lamers, 

Robert Bosch GmbH 

Object Classification exploiting High Level Maps of Intersections 187 

S. Wender, T. Weiss, K. C. J. Dietmayer, University of Ulm 
K. Ch. Furstenberg, IBEO Automobile Sensor GmbH 

Performance of a Time-of-Flight Range Camera for 

Intelligent Vehicle Safety Applications 205 

S. Hsu, S. Acharya, A. Rafii, R. New, Canesta Inc. 



Powertrain 



Coordinated Cylinder Pressure Based Control for 
Reducing Diesel Emissions Dispersion 

M. G. Beasley, R. C. E. Cornwell, M. A. Egginton, R M. Fussey, R. King, 
A. D. Noble, T. Salamon, A. J. Truscott, Ricardo UK Ltd. 

G. Landsmann, General Motors Powertrain Europe 

A High Temperature Floating Gate MOSFET Driver for 
on the Engine Injector Control 

P. Delatte, V. Dessard, G. Picun, O. Stevens, L. Demeus, CISSOID S.A. 



Comfort and HMI 

Distributed Pressure Sensor Based on Electroactive 
Materials for Automotive Application 

E. Ochoteco, J.A. Pomposo, H. Grande, Cidetec 

F. Martinez, G. Obieta, Ikerlan 
J. Lezama , P4Q Electronics 

J. M. Iriondo, Maser Microelectronica 

Intelligent Infrared Comfort Sensors for the 
Automotive Environment 

L. Buydens, V. Kassovsky, R. Diels, Melexis NV 

Gesture Recognition Using Novel Efficient and 
Robust 3D Image Processing 

B. Liu, T. Sunkel, O. Jesorsky, R. Kompe, 3SOFT GmbH 
J. Hornegger, Friedrich-Alexander University, Erlangen-Nuremberg 

Network of Excellence HUMANIST - Human Centred 
Design for Information Society Technologies IN RETS & ERT 

A. Pauzie, French National Institute for Research in Transport and Safety 



223 



239 



249 



261 



281 



295 



Networked Vehicle 



High Speed 1 Gbit/s Video Transmission with 
Fiber Optic Technology 

T. Wipiejewski, F. Ho, W. Hung, S. Cheng, E. Wong, St. Ng, W. Ma, Th. 
Choi, G. Egnisaban, K. Yau, T. Mangente, ASTRI 



Components and Generic Sensor Technologies 

Integrated Giant Magneto Resistors - a new Sensor 
Technology for Automotive Applications 

W. Rossler, J. Zimmer, Th. Bever, K. Prugl, Infineon Technologies AG 
W. Granig, D. Hammerschmidt, E. Katzmaier, Infineon Technologies Austria 
AG 

Simulating Microsystems in the Context of an 
Automotive Drive Application 

G. Pelz, Ch. Decker, D. Metzner, Infineon Technologies AG 

L. Vof3kaemper, D. Dammers, Dolphin Integration 

Miniaturized Wireless Sensors for Automotive Applications 

R. Thomasius, D. Polityko, H. Reich, TU Berlin 

M. Niedermayer, S. Grundmann, S. Guttowski, Fraunhofer IZM 
R. Achterholt, Global Dynamix AG 

3D-MID - Multifunctional Packages for Sensors in 
Automotive Applications 

D. Moser, J. Krause, HARTING Mitronics AG 



307 



323 



343 



353 



369 



PReVENT 



Towards the Automotive HMI of the Future: Mid-Term 

Results of the AIDE Project 379 

J. Engstrom, J. Arfwidsson, Volvo Technology Corporation 

A. Amditis, ICCS 1-Sense Group 

L. Andreone, Centro Ricerche Fiat 

K. Bengler, BMW Group Forschung und Technik GmbH 

R C. Cacciabue, European Commission - Joint Research Centre 
W. Janssen, TNO Human Factors 
H. Kussman Robert Bosch GmbH 
F. Nathan, PSA Peugeot Citroen Automobiles 

Accidentology as a Basis for Requirements and System 

Architecture of Preventive Safety Applications 407 

M. Schulze, DaimlerChrysler AG 

J. Irion, Irion Management Consulting 

T. Makinen, Technical Research Centre of Finland (VTT) 

M. Flament, ERTICO - ITS Europe 

ADAS Horizon - How Digital Maps can contribute to Road Safety 427 

V. Blervaque, ERTICO 

K. Mezger, Daimler Chrysler Research 

L. Beuk, Siemens VDO, Eindhoven 

J. Loewenau, BMW Group Research and Technology 

Intersection Safety - the EC Project INTERSAFE 437 

B. Rossler, Volkswagen AG 

K. Ch. Furstenberg, U. Lages, IBEO Automobile Sensor GmbH 

ProFusion2 - towards a Modular, Robust and Reliable Fusion 
Architecture for Automotive Environment Perception 451 

T. Tatschke, FORWISS, University of Passau 
S.-B. Park, Delphi Electronics Safety 

A. Amditis, A. Polychronopoulos, ICCS, NTUA 

U. Scheunert, Technical University of Chemnitz 
O. Aycard, G RAVI R- 1 MAG INRIA 

Integrating Lateral and Longitudinal Active and 

Preventive Safety Functions 471 

A. Amditis, A. Polychronopoulos, Institute of Comm, and Computer Systems 
A. Sjogren, A. Beutner, Volvo Technology Corporation 

M. Miglietta, A. Saroldi, Centro Ricerche Fiat 



Lane Detection for a Situation Adaptive Lane Keeping 
Support System, the SAFELANE System 

N. Mohler, D. John, M. Voigtlander, Fraunhofer IVI 



Appendices 

Appendix A: List of Contributors 
Appendix B: List of Keywords 



485 



503 

511 



3 



Prospects for MST Sensors in Automotive Applications 

R. Dixon, J. Bouchaud, WTC - Wicht Technologie Consulting 



Abstract 

The number of passenger vehicles on the world's roads will reach an 
estimated 68 million light vehicles in 2008. Many vehicles will fea- 
ture increasingly sophisticated safety and engine management sys- 
tems as a result of the migration of sophisticated MST-based sensors 
from luxury into mid-range cars. The success of such sensors owes 
much to batch micromachining processes, which reduce component 
sizes and provide low power consumption devices that can be inte- 
grated with signal conditioning electronics into small packages that 
handle the harsh environments and deliver reliability extending 
over 10-15 years. 

This paper examines existing and emerging markets for MST sen- 
sors deployed in airbags and vehicle dynamics, GPS navigation, and 
engine management. According to our estimates, the overall market 
for MST based automotive measurements will grow from around 
$1.3 billion in 2004 to nearly $2 billion in 2009. A particular empha- 
sis of this article is the opportunity for inertial and pressure sensors. 



1 Introduction 

Automotive applications demand high-quality products that operate without 
failure, often under extreme conditions. Sensors in this environment must 
combine low cost with high reliability, sustained over production runs amount- 
ing to millions/year. These specifications have continued to be met by 
Microsystem solutions, which have displaced many conventional sensors due 
to a combination of size, low weight and compatibility with the power require- 
ments of existing automotive electronics systems. 

This success explains the increasing migration into less expensive cars. Mid- 
sized vehicles now contain more than 50 sensors, and luxury class cars over 
100 [1]. About one third of these sensors are now MST-based. Meanwhile, the 
automotive sector can now be considered instrumental in the emergence of 
new markets for MST sensors in consumer applications and personal electron- 



ics. 
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The automotive market grows at about 4% annually and various estimates put 
the number of light passenger vehicles as around 68 million by 2008 (e.g. 
Strategy Analytics). Frontal impact airbags and engine monitors that improve 
fuel efficiency and lower emissions have historically driven sensor implemen- 
tation. This is augmented by the migration of advanced safety features such 
as vehicle dynamics systems and anti-roll detectors, in addition to the conven- 
ience of GPS navigation. This continued implementation is expected to contin- 
ue as legislators and consumers propel manufacturers to continually improve 
the safety functionality ride quality and comfort of today's vehicles. 



2 MST Sensor Applications 

An overview of the applications and status for MST sensors are shown in Tab. 
1. Established applications are inertial sensors for vehicle dynamics and 
airbags and pressure sensors for engine management, in addition to flow sen- 
sors for air intake and infrared sensors for a range of functions in traffic/road 
monitoring, cabin temperature control, gas sensor and air quality, anti-fog sys- 
tems, tire and brake monitoring, etc. 



1 Microsystem 


Application 


Status 


Acceleration sensors 


Front airbag, side airbag, upfront sensors, vehicle 
dynamic control, roll detection, GPS navigation, 
active suspension 


Established 


Yaw rate sensors (gyroscopes) 


Vehicle dynamics control, roll detection, GPS 
navigation 


Established 


Pressure sensors 


Air intake (MAP), atmospheric pressure (BAP), fuel 
vapour, turbocharger pressure, tire pressure, HVAC 
compressor, fuel injection, side airbag, passenger 
occupation, oil, diesel particle, coolant 


Established 


Flow sensors 


Air intake, HVAC control 


Established, 2008/2009 


IR Sensors 


Traffic/road monitoring, cabin temperature control, 
gas sensor and air quality, anti-fog systems, tire and 
brake monitoring, seat occupancy 


Established 




Driver vision enhancement 


2008 


Inclinometers 


Headlight levelling, zero setting (gyro), theft alarm, 
electronic parking brake 


Established 


Microphones 


Hands free microphone, directional audio 
microphone arrays 


2008/2009 


RF MEMS switches 


Automotive radar 


2008 



Tab. 1 . Applications and status of various sensors in the automotive sector. 

n engine management, manifold air pressure (MAP) and barometric air pres- 
sure (BAP) sensors represent an entrenched use for silicon micromachined 
pressure sensors. These devices are deformable diaphragms that employ 
piezoresistive or capacitive sensing principles. MAP regulates air intake pres- 
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sure to feedback the correct air-fuel mixture for pollution control and fuel effi- 
ciency and BAP compensates the air-fuel mix according to altitude. 

Fuel vapour monitors and tire-pressure monitoring systems (TPMS) are inter- 
esting applications for pressure sensors. TPMS deployment is driven by the 
legislation in the US, where blowouts and flats reputedly cause 23,000 acci- 
dents and over 500 fatal accidents per year. A typical configuration is repre- 
sented by a pressure sensor (and in many cases an accelerometer to save on 
battery power) placed on each tire that communicates with a central unit via 
RF and displays pressure on the dashboard. 




Fig. 1. Tire pressure-monitor from Infineon and SensoNor (Courtesy of 
Infineon). Most TPMS modules currently contain an MST pressure 
sensor and an accelerometer to save battery power by signalling 
the transmitter to function only when the automobile is in motion. 

MEMS yaw rate sensors and accelerometers are deployed in Electronic 
Stability Program (ESP) modules used for vehicle dynamics and determine the 
correct braking in corners to prevent slip or roll over. Gyroscopes also measure 
the roll of the vehicle and are linked to curtain airbags and seatbelt pre-ten- 
sioning systems if a critical angle is exceeded, and provide dead reckoning 
assistance for GPS navigation systems. Inclinometers are used in headlight lev- 
elling and sensitive alarms. Inclinometers correct tilt in navigation gyroscopes, 
provide sensitive burglar alarms and change headlight level. 

Accelerometers are primarily used as sensors that send signals to fire airbags, 
and today's vehicles regularly feature both side and rear airbags in addition to 
front airbags. Intelligent front airbags have now been designed to respond dif- 
ferently depending on crash severity, while passenger occupancy detection 
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systems (PODS) determine whether it is appropriate to fire an airbag or not. 
Several approaches such as infrared. Hall and pressure sensors compete for 
this application, and are incorporated as a result of low speed incidents that 
triggered early airbag systems and caused fatalities, typically with young pas- 
sengers. These systems are expected to be in all US vehicles by 2009, for 
example. 

In future, the number of different MST sensors used in cars is expected to 
increase, e.g. for infrared devices in vision enhancement, RF MEMS switches 
for automotive radar, and microphones for hands free microphone and direc- 
tional audio microphone arrays. These will be niche applications and not 
expected to represent a significant opportunity for MEMS over the next five 
years. 




2004 

Inclinometer 



2009 

Inclinometer 4% others 

2%^ \ a 3% 

IR Sensor 




15% 

Accelerometer 



$1.29 billion $1.97 billion 

Fig. 2. Major markets for automotive sensors over the period 2004 - 2009. 



3 MST Sensor Markets 

The major markets are broken out by sensor type in Fig. 2. Overall, we fore- 
cast the market to grow from $1.29 billion to $1.97 billion over the period 
2004 to 2009, a CAGR of nearly 9%. Pressure sensors make up the largest 
opportunity in the next five years, followed by gyroscopes for safety applica- 
tions and navigation, airbag accelerometers and flow sensors, mostly for mass 
air flow measurement (and beginning only slowly for accurate HVAC cabin 
control). Inclinometers used in headlight levelling and alarms will not grow 
significantly over the five years and IR sensors will also remain a niche mar- 
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ket opportunity. The remainder of this paper concentrates on the major mar- 
kets for pressure and inertial sensors. 

Pressure Sensors 

Pressure sensors represent the biggest market by revenue - a total of over 
$677 million in 2009 - and will maintain their lead as the largest market 
opportunity by sensor type (see Figure 2). With the exception of TPMS this is 
a mature market featuring sensors for engine management using sensors 
priced from $2.5 to $6 (in 1st level packages). 

The market for pressure sensors used in TPMS will explode to reach $207 mil- 
lion from 90 million units in 2009 (see Tab. 2). This explains the highly com- 
petitive situation and large number of companies producing and developing 
such systems. A TPMS module typically includes pressure sensor, accelerome- 
ter, temperature sensor, signal conditioning electronics, and wireless trans- 
mitter and will cost well under $10 per wheel by 2009. 

After TPMS the biggest market is for manifold and barometric air pressure 
measurement sensors with $175 million, followed by a vast array of under- 
the-bonnet sensor measurements like variable transmission oil and power 
steering applications, or high-pressure fuel injection systems, in coolants, brak- 
ing, and so forth, together worth $157 million. Oil pressure sensors can be 
either ceramic capacitive, silicon strain gauges bonded and fused to mem- 
branes, or piezoresistive silicon chips in silicone oil-filled reservoirs capped by 
stainless steel diaphragms. Oil pressure sensors increasingly use MEMS to 
determine condition as well as level. 





2004 


2009 




ASP 


Units 


Turnover 


ASP 


Units 


Turnover 




[$] 


[millions} 


[$ millions] 


[$] 


[millions] 


[$ millions] 


MAP, BAP (air intake) 


4.20 


36.00 


151.20 


3.50 


50.00 


175.00 


Fuel tank vapour pressure 


5.00 


15.00 


75.00 


4.00 


16.00 


64.00 


Tire pressure monitoring sensor (in 
package, +ASIC) 


2.45 


12.00 


29.40 


2.30 


90.00 


207.00 


Side airbag and passenger 
occupation detection 


3.50 


13.00 


45.50 


2.50 


30.00 


75.00 


Other, e.g. oil, transmission, diesel 
particle, EGR, brake, coolant, HVAC 


6.60 


16.20 


106.92 


4.90 


32.00 


156.80 


compressor 














Total 




92.20 


408.02 




218.00 


677.80 



Tab. 2. Market breakout for MST pressure sensors 2004-2009 

In the area of side airbags, the much smaller crumple zone in car doors 
requires side impact sensors that operate up to 250 g with fast reaction time 
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(few milliseconds). It is worth noting that MST-based pressure sensors are con- 
sidered as alternatives to accelerometers for use in side impact detection sys- 
tems, e.g. from companies like Infineon. Fuel tank vapour detectors are 
mandatory for US markets but represent a saturated market opportunity. 

The major manufacturers of pressure sensors include Bosch, Freescale (with 
foundry partner Dalsa), Denso, Melexis, Silicon Microstructures, Inc., 
Measurement Specialties, Inc. (e.g. crash test accelerometers), GE Nova Sensor, 
and XFAB. VTI planned to introduce pressure sensors in 2005. 

TPMS system suppliers include Schrader with GE Novasensor's piezo-resitive 
pressure sensor, Siemens VDO, Infineon / SensoNor, Freescale, Philips (module 
only) and Bosch, which expects to be on the market in 1-2 years with its solu- 
tion. Many others are in development. 

A recent development is the gradual displacement of bulk micromachining by 
surface micromachining. For example, Bosch recently supplanted its previous 
approach - a silicon chip bonded to a glass socket using anodic wafer bonding 
- by a surface micromachining approach using porous silicon and processing 
steps designed to form a stable cavity. The method lower costs and is designed 
for applications of 10s of bar (metal is used above 100 bar). 

Future developments will include additional demand for high-pressure sen- 
sors, such as required for common rail injection systems and cylinder pressure 
sensors (one per cylinder). The latter will be required for direct pressure deter- 
mination inside the combustion chamber, starting with diesel engines. Lower 
emissions and improved fuel economy are expected to justify the high cost. 
Although the extreme pressure rules out silicon, future in-cylinder sensors 
may alleviate the need for MAP and air intake mass flow devices altogether. 



Inertial Sensors 

Inertial sensors comprise gyroscopes for yaw rate sensing, acceleration detec- 
tors and inclination sensors. Key requirements for the main applications com- 
prise: 

► Vehicle dynamics (ESP) - a yaw rate sensor, typically with a resolution 
of 0.1 °/s, incorporated to mitigate under- and over-steering, and includ- 
ing 1 or sometimes 2 (up to 2 g) single-axis accelerometer(s) with high 
offset stability (100 mg) measure the lateral slip. 

► Roll detection - devices with a resolution of 1 °/s to trigger curtain air 
bags and to pre-tension seat belts in the event of rollover, with an 
accelerometer. 
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► Navigation - dead reckon assistance with a rate grade resolution of 
0.5°/s to assist and improve the accuracy of Global Positioning System, 
sometimes with an accelerometer, but mostly a gyroscope. 

► Tilt sensors (inclinometers) - automatic handbrake, sensitive alarm 
systems, etc. 

The total automotive inertial sensor market will grow from $676 million in 
2004 to $854 million in 2009 (see Table 3). MEMS gyroscopes for ESP and 
accelerometers designed to trigger the inflation of airbags represent the 
largest market opportunities. However, while the number of airbags per vehi- 
cle is increasing, this is not necessarily reflected in the number of sensors, and 
this market will progress modestly in the next few years. ESP systems con- 
taining a 1-axis gyro and acceleration sensor will show healthy growth from 
$280 million to $382 million in five years (CAGR of over 6%). 




Tab. 3. Market breakout MEMS inertial sensors in automotive applications 
2004-2009. * Accelerometers need power and are expected to be 
replaced in more advanced TPMS solutions. * * Highest end vehicles 
only. 



Gyroscopes 

The biggest market for gyros is vehicle dynamics (so-called ESP systems) with 
$315 million expected in 2009. Bosch introduced the ESP system in 1995 and 
has since produced 15 million systems [2]. It leads the market and shipped 4 
million units in 2004, followed by BEI and Silicon Sensing Systems. ESP sys- 
tems are produced in multi-million unit volumes for integrators like Robert 
Bosch, TRW Automotive and Continental Teves (using BEFs gyro). 

A gyroscope module in an ESP system is estimated to comprise around 10-20% 
of the total system cost of $500- $600, which typically includes an MST yaw 
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rate sensor, accelerometer, wheel position sensors, steering wheel sensor, pres- 
sure sensors, control unit, and hydraulic modulator. Depending on how the 
gyro is supplied and the technology, prices range from $15-75. The average 
selling price for an automotive gyro (sensing element, ASIC in a wafer level 1 
package) is estimated to be $20. The price will erode to around $17 or less in 
2009. 

The gyro in a roll detection system does not require the same resolution as 
yaw sensors for vehicle dynamics because roll rates are around five times larg- 
er, although a key requirement is excellent resistance to external shock and 
vibration. Prices average $13 for the gyroscope. 

Navigation systems rely on the use of a compass, databases and a GPS sys- 
tem. When the system is started and the initial direction is established, iner- 
tial data supplied by a gyroscope is used to determine when and how much the 
car has turned, until the direction can be verified and updated via map match- 
ing and GPS signals. There is strong interest in GPS navigation systems in 
Japan, where 2.9 million units - roughly 2/3 of all new vehicles - will be fitted 
with these systems in 2006 [3]. Prices are generally lower, on average $11-12 
for a gyroscope in a navigation unit, because such sensors are non-safety crit- 
ical. 



In terms of operation, gyros determine yaw rate (angular rotation) from the 
Coriolis force, and different manufacturers use different sensing principles. 
The established manufacturers of gyros are Bosch using either bulk or surface 
silicon micromachining to make capacitive measurements of a tethered proof 
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Fig. 3. Composite lithium niobate / ceramic "H" tuning fork gyro element 
for use in navigation and other applications (Courtesy of Fujitsu). 
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mass, BEI Systran Donner (now Schneider Electric) using piezoelectric actua- 
tion and sensing in a quartz tuning fork design, Silicon Sensing Systems 
(Sumitomo/BAE joint venture) with a magnetic solution and Analog Devices 
with a silicon capacitive MEMS design. 

Fujitsu (see Fig. 3), Seiko Epson, Murata, Matsushita, Sony, NEC Tokin and 
Micro Components (owned by Swatch) are examples of other companies pro- 
ducing gyros, particularly for navigation. Piezoelectric materials like quartz, 
lithium niobate or silicon micromachined approaches are popular. OKI expects 
to introduce its gyro by the end of 2006. Gyros are also under development at 
companies looking to break the $10 barrier, such as InvenSense, Samsung, and 
Fraunhofer-IST (marketed by Sensor Dynamics). 

One of the expected trends over the next few years is a consolidation of 
increasing numbers of sensors into multifunctional modules, i.e. clustering of 
inertial sensors to collect information on yaw rate, roll, and for navigation 
dead-reckon assistance. For example, BEI Technologies (part of Schneider 
Electric) has developed a four-degree of freedom module with two gyros and 
two accelerometers. 



Accelerometers 

Accelerometers are used for high g (airbag) and low g (vehicle dynamics, roll 
detect) applications and typically employ silicon micromachining and capaci- 
tive sensing. Airbags will remain the main market for accelerometers ($232 
million in 2009), where devices will cost about $1.50 (wafer level packaged). 
Major manufacturers for high g airbag sensors include Analog Devices, 
Freescale and Denso. VTI and Bosch supply low g sensors, which cost more 
than sensors for airbags (currently $5, dropping to $3.50 in 2009). 
Inclinometers used in tilt measurements are mostly accelerometers operating 
under conditions of static acceleration (Analog Devices, VTI). 

Accelerometers are also used in GPS navigation systems such as Hertz's 
"Never Lost" system, although gyros will prevail for navigation. Tire pressure 
monitoring systems (TMPS) currently feature accelerometers (to inform the RF 
transmitter to signal the tire's pressure only when in motion to save battery 
power) but these will vanish from most systems by 2009 as more intelligent 
low-power solutions are sought. 

Other potential replacement scenarios include body sound ultrasonic sensors 
to replace high-g accelerometers to determine and react appropriately to dif- 
ferent kinds of crash. Siemens VDO is developing "Crash Impact Sound 



12 I Market 



Sensing", intended for introduction in 2007, where supplementary informa- 
tion may be fed to safety systems using camera or radar configurations. 



4 Conclusion 

The market for automotive MST sensors looks bright over the next five years 
with growing sales for sensors that increase efficiency, reduce pollution, 
improve ride safety and provide additional protection in a crash situation. New 
types of sensor will gradually enter the market and enable automotive manu- 
facturers to continually differentiate their product aided by sophisticated low- 
cost MST solutions. 
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Future Architecture for Inertial Sensors in Cars 

J.C. Eloy, M. Potin, Dr. E. Mounier, Dr. P. Roussel, Yole Developpement 

Abstract 

In the analysis of the automotive MEMS inertial market, Yole found 
that on the 2005-2010 period the accelerometer automotive market 
would have a 2% CAGR starting from 286 M$ to321M$. Gyroscopes 
are making good promise on these same period, with a 11% CAGR 
expectation to reach 598M$ in 2010. This year already the gyro- 
scope market overcomes the accelerometer one on a market valued 
at 355M$. 

Two major trends that will modify the future architecture for inertia 
sensors are discussed in this article: the crash impact sound sensor 
technology is subject to change the airbag sensor market whereas 
the trends for integration will lead the path to future automotive 
IMUs. 



1 Automotive Gyros are likely to generate 600M$ Value in 201 0 

Gyroscope applications are today mainly divided in between consumer and 
automotive applications. Automotive represent today 80% of total production 
of gyroscopes with a 22M units market this year. Gyroscopes based automo- 
tive applications shows a high potential and dynamism 

Tab. 1 describes the gyroscope automotive market on the 2005-2010 period. 

Yole's market forecast shows that in 2010 automotive gyroscope will still 
account for 84% of the value generated by total MEMS angular rate produc- 
tion components. It is estimated that the automotive gyroscope market CAGR 
will range from 8% to 30%. 

Three automotive applications are included in our analysis: 

► Anti rollover applications 

► Electronic Stability Control 

► GPS Navigation 
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The main characteristic of the automotive field is that it requires low cost 
chips. The gyros' ASP is in the range $10 to $22, which is still considered as a 
high price for automotive components, thus putting a strong pressure on com- 
ponents manufacturers. We estimate that 54 million of gyros will be necessary 
in 2010 for automotive uses. 





2005 


2010 




Million 

units 


ASP ($) 


Market 

(M$) 


Million 

units 


ASP ($) 


Market 

(M$) 


Roll over (per axis) 


2,5 


10,0 


25,0 


13 


7,2 


93,7 


ESP gyroscope (per axis) 


12 


22,0 


264,0 


25 


15,9 


396,6 


GPS localisation 
(per axis) 


7,5 


9,0 


67,5 


16 


6,8 


107,2 


Other consumer 
applications 


5 




41,0 


27 




117,0 


Total gyroscope market 


27 




398 


80 




714 



Tab. 1. Markets for MEMS-based gyroscope 2005-2010. "Others" applica- 
tions include image stabilization for camcorders, cameras, and 
mobile phones. 



Stability control and other chassis technologies are a very interesting applica- 
tion. This market is truly not the one with the strongest growth expected (8% 
CAGR from 2005 to 2010) but for sure the largest: Stability control will still 
counts for more than 50% of the total gyroscope market in 2010. 

ESC is not a new application and in Europe it stands at a relative mature stage. 
Its growth dynamisms mainly comes from North America. Europe is still 
ahead with an average 40% market penetration against 26% in USA in 2005. 
But we figure out that in one year an average 4% were won on the electronics 
stability control equipment rate in USA. These good results are bounds to the 
promotion it benefited from: 

► Surveys showed that SUV are more subject to loose steering control in 
difficult condition 

► OEM (Original Equipment Manufacturers) are directly promoting their 
systems to the customer 

Anti-rollover systems are likely to benefit as well from these actions. Some 
gyro manufacturers are event expecting a NHTSA regulation to even push 
ahead the pass to full safety equipments features for high centre of gravity 



cars. 
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The performance requirement of chassis control application increase complex- 
ity and competition among players. Few degrees per second bias stability is 
required, depending of the OEM electronics. This is a strong goal to achieve for 
manufacturers. But the 22$ ASP (in 2005) makes it an attractive application. 
We estimate that nearly 90% of the components are supplied by only three 
main challengers in 2005: BEI Technologies, BOSCH and Panasonic. We expect 
an aggressive competition of new entrant in the 2006-2008 periods. 

Gyroscope Manufacturer Market Share on ESC 
application in 2005 



Panasonic 

17% 



BOSCH 

36% 

Fig. 1. Gyroscope Manufacturers' 2005 market share on ESC application 

The year 2008 will be of high interest for the industry. More than 10 players 
would likely offer gyroscopes for stability control and the specific agreement 
between Continental and BEI will end. Only few gyro suppliers will last on this 
application, time will see... 



sss others 

9% 2% 




2 Accelerometers for Automotive Applications: a 320M$ Industry in 
2010 

Acceleration sensors in the automotive industry have been widely used since 
more than 30 years : motion sensing was key in order to be able to assure safe- 
ty functions. But the real development of the market has started in 1988 when 
an un-known Norwegian company named Sensonor has realised on the market 
the first silicon accelerometer able to detect accurately chocks for airbag appli- 
cations. 
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This first application has been really booming during 15 years, with now a 
clear saturation of the market and with key changes we will highlight in this 
article. But during that time, several other applications have emerged. The fol- 
lowing list provides today applications of MEMS accelerometers in the car 
industry. 

► Airbag sensors : detection of front, back and side shock 

► Active suspension : adaptation of the suspension to road condition and 
speed 

► Electronic stability program : system able to control the relative posi- 
tion of the car compared to the road and adjust it in order to be stay in 
safety conditions 

► Safety testing: off road applications, use of accelerometers during crash 
test in order to monitor the acceleration in different points of the car 
and at passengers. 

Tab. 2 presents the status of the different applications: 





Mature 

applications 


Growing 

applications 


Emerging applications 


Accelerometers 


Airbag, 

safety 

testing 


Active 
suspension, 
Electronic 
stability program 
(ESP) 


No new applications are seen at 
the moment. 

GPS is using some times 
accelerometers instead of 
gyroscopes. 



Tab. 2. Status of the different automotive applications 



The total accelerometer market value (see Tab. 3) has been estimated to be 
$355 million in 2005 and $614 million in 2010 for a production of 150M and 
260M units respectively. We estimate that the automotive field accounts for 
89% of the total volumes which generate 81% of the global market value. 140 
million of accelerometers will be necessary in 2006 for the automotive field 
only. 

The major trends that is likely to change the accelerometer business today is 
a technology shift in airbag sensors applications. Siemens VDO soon released 
its Crash Impact Sound Sensing (CISS) sensor in which a MEMS microphone 
will act as a replacement sensor for several accelerometers. 

The sound caused by the car's body panels/structure as they deform under 
impact is used to recognize the severity of the accident. The fact that minor 
collisions and serious collisions have the same rate of deceleration makes it dif- 
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ficult to adapt an airbag system to different crash scenarios. The CISS sensor 
offers strong benefits: 

► Airbag deployment can take place up to 15 milliseconds earlier than 
with conventional systems 

► Directly integrated into the ECU, it might be able sense omni-direction- 
al impacts, thus eliminating the needs for satellite airbag accelerome- 
ters. g crash sensors integrated into the ECU would still be required for 
cross-checking features. 





2005 


2010 




Million 

units 


ASP ($) 


Market 

(M$) 


Million 

units 


ASP ($) 


Market 

(M$) 


Airbag sensor (per axis) 


120 


1,6 


192 


143 


1,3 


182 


Active suspension (per axis) 


1,0 


9,0 


9,0 


1,80 


7,4 


13,3 


ESP acceleration sensor 
(per axis) 


12,0 


7,0 


84 


24,9 


5,0 


125 


Safety related testing 
(per axis) 


0,012 


100,0 


1,2 


0,018 


85,4 


1,5 


Other consumer and medical 
applications 


17,20 




68 


90,435 




292 


Total accelerometer market 


150 




355 


260 




614 



Tab. 3. Markets for MEMS based accelerometers 2005-2010. "Others" 
applications include oil exploration, pedometers, HDD free fall 
detection, auto key-stoning for projections, mobile phone, toys and 
human-computer interface. 

We believe that if this technology is adopted, it would strongly impact 
accelerometers shipments for airbag applications. Its first implementation on 
the Audi A8 2007 model will say if MEMS microphone technology can enter 
the automotive field. 



3 A Path through the Automotive Inertial Measurement Unit? 

Several trends are today very important in the automotive business and are 
impacting very heavily the MEMS business. We will review it in the next para- 
graphs. 
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One sensor for several systems or one system with its own sensor ? 

More and more car manufacturers are pushing the system manufacturers to 
share sensor data between systems in order to save money and complexity in 
the cabling and system architecture. 

One of the key trends is to have a central inertial measurement unit (IMU) able 
to deliver acceleration sensing (3 axis) and gyro movement (3 axis) to all the 
car systems. Several companies like Systron Donner (USA), Bosch (D), 
Honneywell (D), Invensense (USA) ... have or will have devices available very 
soon of automotive IMU. 

The road is long to replace all accelerometers and gyroscopes in a car by a cen- 
tral IMU but the trend is there. Only few companies have the access to the 
right devices, electronics and software to be able to deliver a good and cost 
effective IMU (this is mainly the companies mentioned, Systron Donner, 
Bosch, Honeywell and Invensense). 



Sensor redundancy? 

For several key safety applications like airbag and ESP, the car manufacturers 
are willing to have redundancy for sensors which are key for the safety sys- 
tem, for example for the yaw rate for ESP and the side crash sensor for airbag. 
This trend is becoming very important and also could impact the market on 
the right way : more sensors will be needed, generating more business for all 
the sensor manufacturers. 



Sensor flexibility? 

ECUs have often different location and positions into each car model. This a 
great challenge for OEMs who need to customise systems. As an example, an 
ECU dedicated to stability control and anti-rollover systems needs 250°/s roll 
rate sensitivity whereas stability control yaw sensitivity is 75°/s. 

We identified two ways components manufacturers will solve this issues: to 
develop a specific packaging so that one sensor can be soldered on different 
positions, or ultimately having an 6DOF ECU in which components sensitivity 
and characteristics could be adapted upon the ECU position. 

These three trends are key opposite in their impact on the acceleration sensor 
market for automotive applications: we have tried to take into account these 
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different trends in our market forecasts but the next months and years will be 
very important for the development of acceleration sensing in the automotive 
industry These applications are key for all the players. 



4 Conclusions 

The MEMS inertial sensor markets are widely dominated by automotive appli- 
cations for the years to come and the new applications (both low-end and high- 
end) are driven by the availability of adapted-cost devices with right specifi- 
cations. 2008 year will be a key date for the MEMS inertial industry where 
both gyroscope and accelerometer based applications will be challenged. 
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Yole Developpement 

45, rue Sainte Genevieve 

69006 Lyon 

France 
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phone 



23 



Three-Dimensional CMOS Image Sensor for Pedestrian 
Protection and Collision Mitigation 

P. Mengel, L. Listl, B. Konig, C. Toepfer, M. Pellkofer, Siemens AG 
W. Brockherde, B. Hosticka, O. Elkhalili, O. Schrey, W. Ulfig, Fraunhofer IMS 



Abstract 

Future car safety systems increasingly depend on the availability of 
robust sensors with vital improved technical perception output. For 
this objective, we develop a pulsed laser based time of flight range 
sensor in fully solid-state microsystems technology for more reliable 
detection and classification of road users, vehicles and traffic obsta- 
cles. The development aims on pedestrian protection and mitigation 
of collisions comprising the short distance around vehicle perception 
up to 25 m. The 3D sensor technology based on a chip design of 
64x8 pixels and image repetition rates of up to 100 Hz will be pre- 
sented and the performance of a first 3D line sensor prototype will 
be demonstrated for typical road test scenarios. Furthermore, an 
outlook on the final 3D array camera development for road safety 
applications will be given. 



1 Introduction 

Advanced car safety systems and sensor employment aim to reduce the num- 
bers of road fatalities drastically by providing total awareness to the drivers in 
and around the car and to warn or even actively assist or intervene in situa- 
tions of imminent danger. However, currently available sensors employed for 
the dynamic surveillance of the vehicle environment provide neither the 
degree of reliability and robustness, nor the overall availability of the percep- 
tual output that would be required for safety critical applications. Radar and 
lidar sensors, for instance, yield excellent distance and velocity measurement 
precision, but virtually no lateral resolution. Video cameras feature excellent 
lateral resolution, but the state-of-the-art computer vision processes required 
for scene interpretation are often insufficient to identify all the relevant 
objects depending on contrast variations and environmental illumination con- 
ditions. Laser scanners exhibit much better lateral resolution as compared to 
radar sensors, but they come with comparatively slow scanning repetition 
rates, considerable physical size and comparatively high production costs. 
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In contrast to these drawbacks of the state of the art sensors, the novel pulse- 
operated time of flight (TOF) range sensor approach features detailed distance 
information with lateral resolution at an image repetition rate of up to 100 Hz. 

The benefits for industrial employment comprise: 

► Laser Pulse Based Parallel Time of Flight Method 

► Combination of Precise Range Data with Lateral Resolution ( cp,6) 

► Full Solid State Approach (Standard CMOS), No Mechanical Parts 

► Low Cost, No Maintenance 

► Robust with respect to 

\> Object Surface Reflectivity 
[> Background Illumination (i.e. Sun Light) 

In this paper the principles of the sensor technology and the achieved techni- 
cal data will be demonstrated. Further, the results of comprehensive evalua- 
tion in typical traffic test scenarios will be illustrated and discussed with rele- 
vance to the advantages of machine based 3D perception for semi-autonomous 
braking situations. 

Essential features concern: 

► Faster detection of collision partners 

► Pre-conditioning of the brake in danger situations 

► Application of full braking pressure in emergency situations 

► Short delay, high confidence and high repetition rate of perception 



2 3D Sensor Technology 

The three-dimensional CMOS image sensor is based on a laser pulse-operated 
time of flight method providing parallel distance information to every single 
sensor pixel. In comparison with time-of-flight (TOF) cameras based on the 
principle of CW modulated light emission the pulse-operated method repre- 
sents a unique performance characteristic by featuring vast independence of 
background illumination since laser pulses can be made quite short but nev- 
ertheless keeping the amount of energy per pulse at a constant level. 
Consequently, the background irradiation level is irrelevant as compared to the 
much greater emitted light intensity. With continuously emitted light, this is 
not feasible and the range cameras based on the CW modulated principle suf- 
fer from notoriously poor signal-to-noise ratios as well as of insufficient sup- 
pression of background illumination. 
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2.1 TOF Measuring Principle 

The sensor concept calculates the time of flight (TOF) indirect from the inten- 
sity measurement of single laser pulses in high-speed shutter windows. 
Several laser pulses are averaged on chip to reduce the required laser power 
and to increase the measurement accuracy [1, 2]. The approach is called MDSI 
(Multiple Double Short Time Integration) and will be illustrated by the figure 
below: 




Fig. 1 . MDSI (Multiple Double Short Time Integration) measurement prin- 

ciple 



The transmitted light pulse synchronized with the start of the integration win- 
dow always illuminates the complete 3D scene. The received pulses give rise 
to a linear sensor signal U after the light propagation time T 0 , which is meas- 
ured at two integration times T x (short time shutter) and T z (long time shut- 
ter). While in the short time shutter only a certain fraction of the light pulse is 
detected depending on the distance of the object point, the long time shutter 
always receives the full reflected light intensity. By computing the quotient of 
the two integrated shutter intensities U x ! U z an exact calculation of the propa- 
gation time T 0 and hence the distance d of each individual object point can be 
derived according to a simple mathematical relationship (see Fig. 1). 

A unique feature of the MDSI method is the analogue real-time on-chip accu- 
mulation process and correlated double sampling procedure (CDS) at each indi- 
vidual sensor element using multiple pulses for each shutter window. The 
amount of received light reflected from an object not only depends on the 
emitted laser irradiance, the reflectance of the object and its distance, but also 
on the amount of background light due to effects of other present light sources. 
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i.e., sunlight. The influence of background light can be widely eliminated by 
measuring solely the background irradiance without laser pulse. Thus each 
measurement must be performed with laser pulse on and off and only the dif- 
ference being stored on the corresponding capacitance in the analogue sensor 
memory. By this means the signal-to-noise ratio increases and leads to an 
improved distance accuracy. Repetitive integration of laser pulses will be per- 
formed adaptive up to the saturation level at each pixel element. Intelligent 
procedures can be employed to cover a large dynamic range of different sur- 
face reflectivity of targets with this adaptive flash illumination method. 



2.2 Chip Design and Functionality 

The 3D CMOS sensor chip contains a photodiode array and readout and con- 
trol electronics. It employs the multiple double short-time integration (MDSI) 
principle mentioned above. This time-of-flight (TOF) approach ensures the best 
background light suppression, non-ambiguity and, using the adaptive inte- 
gration mode, a very high dynamic range. Adaptive integration means multi- 
ple non-destructive readout of the accumulated signal during integration of 
the reflected laser pulse train. The sensor allows on-chip analogue integration 
of more than 128 pulses. 



Parameter 


Specification 


Pixel count /size 


64x8=512 / 130pm x 300pm 


Pixel principle 


Photodiode with single buffer and CDS 


Noise equivalent power (30ns) 


1 W/m 2 (Design goal) 


Minimum shutter width 


30 ns 


Dynamic input range 


54 dB + 21 dB (variable # of pulses) 


Image acquisition 


50. ..200 fps (1...128 pulses/frame) 


Readout scheme 


Non- destructive readout 


Background light suppression 


>40 dB 


Operating wavelength 


850 nm...910 nm 



Tab. 1 . Sensor chip parameters 
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The most challenging sensor specifications are the optical dynamic input 
range (DR) and the noise performance (NEP). The DR is determined by the dis- 
tance range and the reflectivity range (min/max). With a given optics, the 
received optical signal decreases with the square of the object distance. Thus, 
the required distance range of 2 m - 25 m and reflectivity range of 5% - 100% 
call for a minimum DR of 70 dB. 

The number of photons reflected by the objects in the scene back into the sen- 
sor lens is quite low. Therefore, minimizing sensor noise is of prominent impor- 
tance. Careful photodiode and readout electronics design and detailed noise 
analysis and optimization yielded a significant reduction in noise equivalent 
power, NEP. The key sensor specification data is summarized in the Tab. 1 . 

The sensor chip has been designed in the standard 0.5 pm CMOS process 
(1P3M) of Fraunhofer IMS with no process modifications. The mask layout of 
the CMOS sensor is shown in Fig. 2. The chip dimensions are 11 mm x 9 mm, 
and the photosensitive area in the centre measures 8.4 mm x 2.4 mm. The 
chip is mounted in a standard ceramic CQFJ84 package. 




Fig. 2. Mask layout of 64x8 pixel CMOS TOF sensor chip 



2.3 3D Test System 

A first prototype system has been established for validation of the sensor con- 
cept in typical road safety related test scenarios. The prototype consists of an 
intermediate CMOS line sensor with 64 x 1 photosensitive elements, appropri- 
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ate designed electronics, optics and the laser pulse illumination. An image of 
the test system is shown in Fig. 3. 




Fig. 3. Prototype test system 



The design of the sensor electronics is based on a multi-board architecture to 
provide high flexibility for operation in different applications. Fig. 4. The main 
board contains the 3D-CMOS sensor, ADC-analogue-to-digital conversion and 
an integrated FPGA platform for sensor control, data acquisition and image 
processing, respectively. For noise optimization, special emphasis has been put 
to the placement of analogue and digital components and to the selection and 
design of voltage regulators. The interface board includes a 1 2 V power supply 
and various application interfaces as RS 232, TBaselOO Ethernet and User I/O. 
The synchronisation to the laser source is realised by a trigger timing signal 
from the FPGA to the electronic driver circuit of the laser module. 




Fig. 4. Sensor system design 
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The laser modules with integrated driver electronics fulfil the high require- 
ments on output power and variable pulse width with fast rise time. Thus, 
adjustable pulse widths between 100 ns and 300 ns with rise times of a few 
tens of ns can be reached with energy of 75 pWs at 10 kHz pulse repetition 
rate. Special beam forming optics has been constructed with custom designed 
cylindrical lens arrangements. According to the application requirements the 
illuminating laser light source can be formed with a homogeneous or 
anisotropic beam profile corresponding to the scene and the cameras field of 
view. 



3 Road Safety Applications 

The intended road safety applications and recognition tasks suitable for the 3D 
CMOS image sensor comprise: 

► Near to intermediate front and side distance range of cars (20 m - 25 m) 

► Lateral proximity of trucks (blind spot surveillance) 

► 3D-algorithms for object (obstacles, pedestrian) detection and classifica- 
tion independently of surface reflectivity and ambient illumination con- 
ditions 

► Performance investigations with car/truck demonstrators on typical 
traffic scenarios and obstacle occurrence 

The application specific requirements for the 3D camera system have been 
analysed considering various traffic scenarios and relevant accident statistics 
[3]. From these investigations the essential consideration of pedestrian protec- 
tion and collision mitigation will be demonstrated in the following chapters. 



3.1 Accident Analysis between Passenger Cars and Pedestrians 

Most front-end collisions with pedestrians occur in urban traffic situations. 
Therefore, the speed of the involved vehicles is typically 50 km/h or less. 
Walking pedestrians reach a characteristic speed of up to 6 km/h. It is assumed 
that approximately 0.5 seconds will be needed to detect a pedestrian in front 
of the vehicle and an additional 0.5 seconds for preconditioning of the brakes 
to achieve full braking performance. The required observation range of the 3D 
camera is shown in Fig. 5. 

Pedestrians outside the blue lines are not at risk of a collision with the vehicle. 
A vehicle travelling with 50 km/h has a stopping distance of about 23.5 meters 
and therefore requires a longitudinal measurement range of 25.5 meters. A 
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pedestrian can walk about 3.6 meters within the stopping time of the vehicle 
and therefore requires a lateral measurement range of 4.5 meters. For lower 
vehicle speeds the stopping time and distance becomes smaller and also the 
walking distance of the pedestrian becomes smaller. Therefore, the required 
measurement range of the camera is much smaller for lower speeds but the 
required aperture angle becomes larger because of the smaller relative speed 
between vehicle and crossing pedestrian. 




Fig. 5. Required observation range of the 3D camera 

The relationship between the required measurement range and the measure- 
ment direction is shown in Fig. 6. For a vehicle speed of 50 km/h, the required 
measurement distance range is up to 26 meters at an aperture angle of up to 
10°. For lower speeds the required measurement range becomes smaller but 
the aperture angle increases. For a vehicle speed of 10 km/h the required 
measurement distance range is up to 6 meters at an aperture angle of up to 
±30°. 

From the considerations regarding the front-end collisions with crossing 
pedestrians we can derive the following requirements for the 3D camera 
measurement range: 

Minimum distance: 2 m (vehicle front end) 

Maximum distance: 26 m for a = + 10° (v = 50 km/h) 

6 m for a = + 30° (v = 10 km/h) 
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This allows the design of an optimised beam shape for the active laser illumi- 
nation of the 3D camera. 




3.2 Front- to rear-end Collision caused by Passenger Vehicles 

Like with accidents between cars and pedestrians, the geometry of the per- 
ception field required for the mitigation of front- to rear-end collisions between 
vehicles is characterised by a comparatively narrow strip parallel to the axis of 
the trailing vehicle. As with car-pedestrian encounters, a comparatively short 
distance range of a few meters is sufficient at an aperture angle of 30 degrees, 
while the maximum distance range is required at the central lines of sight 
close to the vehicle axis. This anisotropy can be used again to redistribute the 
emitted laser power accordingly, thereby gaining distance precision and/or dis- 
tance range in the central field of view. Concerning the kinematics of the rear- 
end crash scenarios, one has to consider the distance, the relative velocity and 
the relative acceleration of the potential collision partners. Collision avoidance 
thus requires appropriate time to collision estimates, which in turn are based 
on appropriate measurements of the relative speed and deceleration. With a 
relative precision of distance measurements of 20 cm or less and a measure- 
ment repetition rate of 50 to 100 Hz, the 3D camera is in a good position to sat- 
isfy these essential features under the assumption of constant deceleration 
conditions. If this is not the case, the forecast of the time-to-collision and/or of 
the crash velocities becomes difficult to predict at any rate. 
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It can be shown that a machine-based accident avoidance / accident mitigation 
system can clearly outperform the average human driver. This is so, despite of 
the fact that the system is based on a perception device of some limited dis- 
tance range of only about 20 m to 30 m. The outstanding performance of the 
mitigation system as compared to human drivers is rather based on the advan- 
tages of: 

► Some best possible system response / delay times of only 0.5 s or less, 

► The employment of maximum deceleration value of up to 10m/s 2 and 

► A perception device that produces virtually no false alarms. 

The 3D range camera appears to be a superior candidate for the type of acci- 
dent mitigation perception device proposed in this section. 



4 Car Evaluation Results 

The 3D test sensor system has been integrated in an experimental car for eval- 
uation on road safety related test scenarios. 

4.1 Technical Data 

The performance data are summarized in Tab. 2. 



Parameter 


Value 


Pixels: 


64 x 1 


Pixel size: 


(130x600) (xm 2 


Laser pulse energy: 


75 |xWs/ 150 |xWs 


Pulse width: 


300 ns 


Pulse frequency: 


5 kHz 


Opening angle: 


36°/ 60° 


Frame rate: 


theoretically, 25 fps @100 integrations (not reached due to 
data transfer and computing time of Matlab application) 



Tab. 2. Performance data of the test sensor system 

A typical distance calibration curve is represented in Fig. 7 for a target with 
5% reflectivity (black velvet). Averaging over 10 single measurements is 
applied to lower the noise of the measured distance. The system was equipped 
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with an anisitropic illumination unit with pulse energy of 1 50 pWs per pulse 
and optics with 36° mm opening angle. 




As can be seen from the measurements the sensor system provides high lin- 
earity between measured and real distance over the entire measurement 
range. The measuring accuracy i.e. the standard deviation of the measure- 
ment results for one point on the curve, result in 5% of distance reading. 



4.2 Road Safety related Test Scenarios 

Different test scenarios were evaluated in order to investigate the ability of the 
test sensor to fulfil the requirements of the above mentioned road safety appli- 
cation. See Tab. 3 for an overview over the test scenarios and the particular 
conditions. For all following measurements, imaging optics with 8 mm focal 
length, resulting in an opening angle of about 60°, was used. 
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Test Scenario Situations 




Scenario 


Target 


Laser 


Theoretical 

image 

repetition rate 


Target 

Distance 


Example Picture 


A 


Frontal view 


Vehicles 


E=75pWs 


4 fps 


5 m / 1 1 m 





from car 



isotropic 

illumination 



B Frontal view Pedestrian E=150pWs 6 fps 
from car anisotropic 

illumination 



C Blind spot 
surveillance for 
a truck 



Pedestrian E=150pWs 16 fps 
anisotropic 
illumination 



16 m 



12 m 



<2m 



Tab. 3. Test scenario descriptions 



4.2.1 Scenario A: Vehicle Detection 

In the test scenario, both cars (Renault and Mercedes) are visible to the test 
camera. The left figure shows the traffic scene taken with a photograph, the 
right figure shows the sensor response (shutter 1 in red, shutter 2 in yellow) 
and the estimated range data (in blue). The estimated range of 1 1 m and 5 m 
complies well with the actual situation. In the gap between both cars the 
reflected signal drops below the noise threshold, and thus yields no usable 
range data. This is probably due to the inclined lateral surface of the Mercedes. 
The metallic varnish has non-lambertian reflection properties. The depth pro- 
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file of the cars themselves is given by pixels 38 to 42 for the Mercedes and pix- 
els 48 to 56 for the Renault. However, the depth variations in these regions are 
strongly influenced by (partly systematic) depth noise of about 0.3 m. But even 
though the individual obstacles have this noisy depth profile, the overall situ- 
ation is well observable in the range data. 




Mercedes at 1 1 m 



Renault at 5m 



pixel 



Fig. 8. Two cars in front 



4.2.2 Scenario B: Pedestrian Detection 

This scenario (Fig. 9) shows a pedestrian walking towards the car. For a dis- 
tance of 18 m the sensor response (red and yellow line) is quite low, but though 
an accurate distance measurement is performed. The width of the pedestrian 
in the range image is only three pixels. At 1 2 m the distance measurement is 
still very accurate, the width of the person increased to 5 pixels. At a distance 
of 8 m one observes a margin effect: the pixel at the right boundary of the 
pedestrian only gets a partial reflection, thus the estimated depth is too high. 
Nevertheless, for the other pixel also here the measured distance fits well with 
the real distance. Thus it can be seen that the sensor provides good measuring 
linearity over the whole distance range. 



4.2.3 Scenario C: Blind Spot Surveillance 

This scenario (Fig. 10) shows a pedestrian walking towards the car. For a dis- 
tance of 18 m the sensor response (red and yellow line) is quite low, but though 
an accurate distance measurement is performed. The width of the pedestrian 
in the range image is only three pixels. At 1 2 m the distance measurement is 
still very accurate, the width of the person increased to 5 pixels. At a distance 
of 8 m one observes a margin effect: the pixel at the right boundary of the 




36 I Safety 



pedestrian only gets a partial reflection, thus the estimated depth is too high. 
Nevertheless, for the other pixel also here the measured distance fits well with 
the real distance. Thus it can be seen that the sensor provides good measuring 
linearity over the whole distance range. 




Fig. 9. Sequence of pedestrian approaching the test car 

Many severe road accidents at crossings, junctions or roundabouts are caused 
by truck drivers who are unaware that other road users are very close to, or 
beside their vehicles. This is illustrated in the above scenario where a child is 
crossing the street directly in front of the truck. As seen from the overlaid 3D- 
profile the safety of vulnerable road users can be improved considerably in 
such situations with direct 3D distance measurements. 

Depending on the mounting position, of the sensor blind spot detection 
requires an opening angle of the sensor from 60° to 90°. 
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Fig 10. Blind spot surveillance 

4.3 Conclusion 

The current development platform shows the ability of the test camera to 
resolve the car proximity at distances of up to about 20 m. The sensor per- 
formance was successfully investigated in different test scenarios, including 
moving objects. 

However, several areas have been identified that call for enhancement: 

► Vertical Range: Currently, only a single line of the camera is functional. 
This limits the vertical field of view to about 2°. Thus, if the car 
approaches an object of small vertical size, it can disappear in the range 
data. The 64x8 - sensor (cf. 2.2) is necessary to overcome this problem. 

► Frame rate: The current restrictions in illumination, sensitivity, serial 
communication and visualisation limit the frame rate to about 5 fps. 
Enhancements in all of these areas must strive for a frame rate of at 
least 50 fps to allow a reliable tracking of traffic objects. 

► IR transparent windows: The car tests showed that the thermal insula- 
tion of the windscreen blocks the IR laser illumination. Thus the test car 
needs either a windscreen that is not insulated, or a cut-out area with- 
out IR block coating. 
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► Opening angle: At distances over 20 m a pedestrian would be repre- 
sented by only one or two pixels. Thus, an object classification will not 
be possible at these distances due to the low resolution. Reducing the 
field of view for front view applications from 60° to 30° would double 
the number of pixels per object and still provide the necessary field of 
view for the intended tasks. 



5 Outlook 

The evaluation results showed that the camera offers outstanding potential not 
only in view of the required perception performance, but also with respect to 
sensor size, power consumption, package requirements and future production 
costs. In addition we expect the following performance enhancement for the 
final 3D range camera in the future project developments: 

► Establishment of a CMOS array with 64x8 pixels providing lateral reso- 
lution in two directions instead of a line sensor 

► Improvement of the sensor sensitivity of a factor of 2 (noise equivalent 
power NEP = 1 W/m 2 ) 

► Implementation of laser modules with higher output energy and high- 
er repetition frequency (30 kHz). The laser illumination will still fulfil 
laser class 1 requirements 

► Employment of a customised optics with lower f-number (f # < 1) and a 
field of view of 30° to increase irradiance at sensor level, as well as 
scene resolution at larger distances for object classification 

The planned modifications will result in improvements concerning distance 
range, accuracy of distance measurements and frequency of image acquisition. 
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Abstract 

Floating Car Data is a well known technology used in traffic science 
to detect travel time, average speed and disturbances with the help 
of probe cars moving as mobile sensors in a road network of inter- 
est. As GPS and GSM is available all over Europe, it is nowadays pos- 
sible to generate FCD data with a simple software implementation 
on mobile devices such as PDAs, MDA's or navigation systems con- 
nected to a GSM module. After the generation of FCD based traffic 
data it can be sent to a service centre and help to improve fleet and 
traffic management. Results from field trials in Berlin, Hanover and 
Athens will be presented. 

There are three different approaches to set up an FCD fleet for traffic man- 
agement: 

1. Commercial fleets with an on board GPS and communication system, 
e.g. caps or trucks. 

2. Private or commercial fleets with an on board unit including GPS and 
GSM, e.g. navigation systems with an integrated mobile phone. 

3. Public transport or shuttle fleets using a number of different technolo- 
gies for the location of the fleet and for the communication to the cen- 
tre where delay time and schedule have to be supervised. 

Since 1990 all of these FCD approaches have been analysed within a number 
of research projects but costs and benefits were not balanced enough to run a 
commercial service centre based on FCD technology alone. In order to find a 
cost efficient FCD service concept gedas, Signalbau Huber and FHG-IPK devel- 
oped a system which allows the control of costs and benefits, including imple- 
mentation and operation of the service centre. 

The idea is to equip public transport and shuttle fleets (see Fig. 1) in order to 
receive scheduled data in the service centre. After the service centre knows 
that parts of the road network are congested the fleet will be informed auto- 
matically and a deviation recommendation will be announced. This scheduled 
FCD approach allows an optimisation of fleet and traffic management during 



42 I Safety 



large scale events and gives a reliable source of "mobile" traffic data to the 
organising entities responsible for schedule and security. The technology was 
developed within the "Eye in the Sky" project and tested in Berlin, Hanover 
and Athens. The vehicle and VIP sponsor of large scale events, e.g. Olympic 
Games, usually equips his fleet with mobile phones and navigation systems, 
thus the technology can be implemented with a simple software update. 

As the Volkswagen Group was chosen by the National Olympic Committee of 
China to be the official auto sponsor for the Olympic Games in Beijing 2008, 
gedas will continue the first experiences of the "Eye in the Sky" tests in 
Germany and Athens to validate a possible transfer towards the Olympic 
Games 2008 in China. First step will be an FCD implementation within the 
traffic management centre of Hanover to support the park guidance system 
during the FIFA World Cup in Germany next year. The project started 2004 
and is co-financed by the ministry of transport of the government of 
Niedersachsen. 

The first live test of the "Eye in the Sky" high-tech system was presented the 
1 st of July in Berlin where a German consortium including the DLR (Deutsches 
Zentrum fur Luft- und Raumfahrt, German Aerospace Center), Fraunhofer IPK 
(Institut fur Produktionsanlagen und Konstruktionstechnik, Institute for 
Production Systems and Design Technology), the Bosch subsidiary Blaupunkt, 
ND SatCom, and gedas demonstrated the integrated "Eye in the Sky" system 
of airborne and terrestrial traffic detection. The "Eye in the Sky" project, a 
four million euros project funded by the EU, combines two different traffic 
monitoring strategies: capturing traffic data with the gedas CityFCD (Floating 
Car Data) and the local monitoring from a helicopter using special DLR cam- 
eras. 

The resulting solution was breaking new ground. It offers a unique approach 
to improving security at major large scale events such as the Olympics or 
world championships. Real-time data capture is the key to enabling the smooth 
deployment of the police, emergency service vehicles and to supporting spon- 
sors and organizers in their operations. The "Eye in the Sky" will allow traffic 
control centres to plan with more precision and to provide more accurate rec- 
ommendations if the traffic threatens to become jammed or an emergency 
occurs. In addition, an ND SatCom antenna in the helicopter provides an 
independent radio channel for the police and emergency services. Such a safe- 
guard is an invaluable aid in earthquake-prone areas or for preventing terror- 
ist strikes. Vehicles equipped with CityFCD and gedas software implemented 
on Blaupunkt hardware were the core component of the live test in Berlin. 
They acted as mobile sensors on the streets. The data gathered from the "flow- 
ing fleet" were transmitted via SMS text message to the prototypical traffic 
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control centre at Fraunhofer IPK, where they were immediately evaluated on 
digital road maps. If the data suggested an upcoming traffic jam, the "Eye in 
the Sky" came into action - and the helicopter flew over the potential bottle- 
neck. Scanning the streets from above, the special high-resolution DLR cam- 
eras within the helicopter fed the traffic control centre with image data via the 
ND SatCom radio connection. The data were immediately evaluated and made 
available to the traffic control centre. In effect, the system not only visualised 
but even measured the traffic volume at key traffic nodes. The advantage of 
this holistic solution is that it yields a high-quality, real-time data pool, which 
helps traffic controllers make the right decisions. 




Fig. 1 . System overview 
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Advanced Pressure Sensors with high Flexibility for Side 
Crash Detection 
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Abstract 

The implementation of advanced regulations for side impact crash 
tests require an improved and faster detection method for side 
impact crashes. Therefore the sensing principle must recognize the 
severeness of an accident with highest possible confidence and with- 
in shortest time. The use of pressure sensors inside a door cavity 
enables the system supplier to develop solutions to fulfill these 
requirements. Infineons 3 rd generation of pressure sensors for side 
crash detection offers a high flexibility to implement customer spe- 
cific communication protocols. Furthermore, proven diagnosis func- 
tionality as well as additionally new features are integrated to guar- 
antee a high reliability of the sensors for this special safety applica- 
tion. All active components for the entire pressure satellite could be 
integrated within the sensor. 



1 Introduction 

Due to increased traffic accidents by side impact crashes and from it devoted 
rising number of hurt passengers of a car was intensified the guidelines for 
execution by side impact tests. Here increased requirements result to detect- 
ing such side impact crashes. The sensor system which can be used must rec- 
ognize the heavy of the accident surely and within shortest time. The acceler- 
ation sensors used for this task show here however serious disadvantages: 
They can take up an impact by their preferential installation way in the b-pil- 
lar of a vehicle on the vehicle door only delayed. The use of a pressure sensor 
makes possible completely different beginning for detecting a side crash. 
Since by a side impact on a vehicle door this is distorted, an increase of pres- 
sure results within the door cavity. This pressure impulse can be measured by 
pressure sensors, which are in the door cavity. This pressure sensor was par- 
ticularly developed for this dedicated application and is characterized by a 
high device complexity and many years proven high reliability. 
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2 Side Impact Tests 

Conventionally, the tests for side impact crash were performed with a barrier, 
which meets the vehicle doors and the B-pillar from the size both parallel to 
the vehicle side (ECE-R95, 96/27/EG, euro NCAP, IIHS and FMVSS 214, Fig. 1). 



I M— 
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• Movable deformable barrier 

• Speed: 50 km/h 

• Angel: 90° 



• Movable deformable barrier 

• Speed: 48.3 km/h (x-direction) 

24.1 km/h (y-direction) 

• Angel: 90° / 63° 



Fig. 1 . Conventional side impact tests 



Thereby the impact impulse was passed on directly to the B-pillar of the vehi- 
cle. An acceleration sensor located in the B-pillar can detect this impact suffi- 
ciently fast, in order to give the information for the release of the safety sys- 
tems. By the rising license number of SUV (sport utility vehicle) and the high- 
er type of construction of these vehicles it comes in the case of a side impact 
crash by such a vehicle however to an accident picture, which is not suffi- 
ciently covered by the conventional side impact tests. Accidents with such 
vehicles only met the vehicle door and not the entire vehicle side and thus the 
B-pillar. The consequence from these accident pictures is an advanced guide- 
line for side impact tests (FMVSS 214 NPRM, Fig. 2). 




• Pole Impact 

• Speed: 29 km/h 

• Angel: 90 0 




• Oblique Pole Impact (FMVSS 214 NPRM) 

• Speed: 32 km/h 

• Angel: 75 0 

• New test criteria because of increased accidents 
with SUV/LTV 



Fig. 2. Advanced side impact tests 
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3 Operational Principle 

The guidelines for the advanced side impact tests, changed on basis of char- 
acteristic accident pictures, show an impact by means of a pole within the 
vehicle door. In that case the B-pillar is not directly affected by the impact, 
whereby acceleration sensors that are located at the B-pillar, can detect the 
collision due to inertia only delayed. 




Fig. 3. Placement of the pressure sensor within a vehicle door 

Measuring the impact at the place of the happening offers different detection 
possibilities. In the case of a sufficiently strong impact on a vehicle door it 
comes to the deformation. This causes a pressure impulse within the door cav- 
ity, since primarily the exterior of the door is pressed in the inside, the door 
inside paneling however a spontaneous pressure balance retarded. The pres- 
sure sensors developed particularly by Infineon for this application are located 
within the door cavity (Fig. 3). They are possible to detect the pressure 
impulse during an impact. 

Fig. 4 shows the output signals represented by pressure sensors and accelera- 
tion sensors during a side crash impact. In each case two events are regarded: 
An impact is present after FMVSS 214, whereby the safety systems must be 
activated and an innocent impact of a ball is present, whereby safety systems 
may not be activated. 
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Fig. 4. Pressure sensor signals compared with acceleration sensors 

Fig. 4 shows the advantages of the pressure sensors here clearly: On the one 
hand the reliable decision to activate the safety systems is know importantly 
faster. On the other hand pressure sensors show an output signal wherewith 
it is substantially easier to differ between material accident and insignificant 
impulses. 

A further advantage when using pressure sensors for side impact detection is 
that by the constant pressure distribution in the door interior the entire door 
serves as sensitive element. Thus the output signal is independent of in which 
place of the door the impact takes place, but only of the strength of the impact. 
From this a renewed advantage results concerning the placement within the 
door and the mounting technique: The output signal of the pressure sensor is 
of it independent. Further the pressure sensor will only then supply relevant 
initial values if it comes to an active destruction of the door; a wrong decision 
can be minimized in such a way. An interesting point is besides the direct link- 
age between impact strength and output signal of the pressure sensor. Here 
safety systems are conceivable, which can be activated gradually depending 
upon accident strength. 



4 Special Pressure Sensors for Side Crash Detection 

Already since 1998 with Infineon special pressure sensors are successfully 
produced for side crash detection, which today already find in many current 
passenger car platforms use. The improved guidelines for side impact tests 
show an intensified interest in this application. Based on no standard in the 
data communication but the necessity of each module manufacturer to have 
an adapted sensor for his system, larger product flexibility is in demand. Due 
to this request, Infineon develops the third generation of special pressure sen- 
sors for side crash detection. The higher flexibility for the customized trans- 
mission protocols could be achieved by the integration of an intelligent state 
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machine (iSM). Contrary to a fixed state machine, whose program sequence is 
firmly defined, the iSM gets its instructions from a firmware. This firmware is 
deposited within an integrated ROM. Changes mean only a change of the 
ROM. The used iSM already worked with several sensors from Infineon. Apart 
from the new flexibility it applies to receive however the proven reliability of 
these sensors. For this within the sensor several diagnosis modes are integrat- 
ed. 







The sensor transfers not the absolute pressure, but the application adapted 
relative pressure ( AP/P 0 ). This is calculated by the difference of the absolute 
pressure with the ambient pressure, standardized on the ambient pressure 
(Fig. 5). The ambient pressure is produced thereby by means of a low-pass fil- 
ter from the current pressure. In order to receive reliability over the correct 
function of the sigma delta converter and the following filters, during the 
starting phase the converter is supplied with a constant voltage and the sys- 
tem response of the filter is compared with the expectancy value. In the case 
of a deviation an error code is sent. The detection of the pressure is not made 
by a single large pressure cell, but within the sensor several pressure cells are 
integrated. 42 of such cells forms a pressure field, whereby altogether 2 sen- 
sitive pressure fields and 2 reference pressure fields are present (Fig. 6). 
During the normal operation these pressure fields are connected within a 
Wheatstone bridge. During the start-up phase the pressure fields are connect- 
ed in such a way that a direct comparison of the sensitive pressure fields 
becomes possible. Thereby damages of the pressure cells can be recognized. In 
that case an error code is sent to the airbag control unit (ECU), which is able 
to shut down only the defect sensor. Pressure cells with only one single pres- 
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sure cell can provide this test only with the help of a reference sensor (right 
door - left door). In the case of an error both satellites would have to be driven 
down with such systems. 




Fig. 6. SEM picture of the pressure cells in comparison to the head of an 
ant 



Additional diagnosis modes like a self check iSM and the permanent monitor- 
ing of the ambient pressure are further characteristics of Infineons third gen- 
eration pressure sensors for side crash detection. 

Furthermore, these sensors feature a high integration of the components, 
which are needed for pressure based side crash detection satellites. Only few 
additional passive elements are needed (Fig. 7). Thereby costs can be saved 
regarding the application. 
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Fig. 7. 



Application circuit 
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Summary 

The third generation pressure sensors for side crash detection is characterized 
by a large flexibility regarding the customer requirements for specific proto- 
cols. Furthermore, proven diagnosis modes as well as additionally new modes 
are integrated. Thereby a further high reliability of the sensors for this special 
safety application can be guaranteed. Additionally all active components for 
the entire application could be integrated within the sensor. This high device 
complexity makes a cost reduction possible on module level. 
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Abstract 

This paper deals with a novel sensor fusion approach to detect and 
track cars and pedestrians to facilitate a collision mitigation applica- 
tion for vehicles. Robust collision mitigation requires a perception 
performance of an unprecedented degree of reliability, since an erro- 
neous application of emergency braking caused by false alarms 
would greatly impede road safety improvement not lastly due to the 
major setback such an incident would represent for driver accept- 
ance. However, current off-the-shelf single sensor approaches can 
hardly fulfil the challenging demands. Accordingly, we develop a 
multi-sensor recognition system. It is composed of a far infrared 
imaging device, a laser scanner and several radar sensors, which 
operate integrated into a BMW sedan. 



1 Introduction 

Statistic evidence of the European Union shows that accidents resulting in 
fatalities or serious injuries are caused to the highest percentage by collisions 
of cars with vulnerable road users. This fact points to the urgent need for 
active and passive automotive safety systems as a significant contribution to 
overall road safety. 

Focusing on a novel approach for environmental perception based on a multi 
sensor-system this paper offers a collision mitigation application for cars by 
means of autonomous braking. To meet the application's requirements regard- 
ing accuracy and reliability of the perception result, we propose a fusion pro- 
cessing scheme, which operates only on slightly pre-processed sensor data. 
This "early fusion" approach uses the synergetic effect of a common and con- 
sistent data processing as well as an interpretation of sensor low-level data to 
tap almost the full sensor potential. To this end it relies on a combined model- 
ling of the environment, which contains object assumptions as well as a-priori 
knowledge. 
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1 .1 Related Work 

Many research groups have contributed significantly in the field of sensor 
fusion. With regard to an automotive environment Kampchen et al. [6] propose 
an adaptive cruise control system (ACC) based on a combination of an early 
and a track-based fusion approach. A laser scanning sensor and an imaging 
camera are used to detect vehicles. Schweiger et al. [8] utilize a radar and a 
monocular imaging sensor to build an ACC system. A collision warning and 
vision enhancement system is proposed by Polychronopoulos et al. [7]. 
Vulnerable road users and vehicles are identified by a far infrared camera and 
a radar sensor. 

In this paper the multi-sensor perception system is composed of four radar 
sensors, a laser scanning device and a far infrared camera to detect both vul- 
nerable road users and other vehicles utilizing a novel early fusion approach. 



1 .2 Overview 

This publication focuses on a novel fusion approach based on only slightly pre- 
processed sensor data. The given sensor platform and its configuration is dis- 
cussed in chapter 2. The envisaged safety application on top of the perception 
system is presented in chapter 3. Chapter 4 is dedicated to the perception sys- 
tem. After a short motivation with respect to the early fusion concept (4.1), 
section 4.2 is giving an overview on the fusion cycle and the system's main 
components. In the following sections (4.2.1 to 4.2.3) the data acquisition as 
well as the time prediction of our system is introduced, and the generation of 
predicted measurements is explained in detail. Sections 4.2.4 and 4.2.5 illus- 
trate the data association process and the generation of new object hypothe- 
ses within the proposed fusion framework before the filtering of the object 
information is addressed (section 4.2.6). Finally, the last section gives an 
overview on the system architecture and implementation details of the fusion 
system. 



2 Sensor Configuration 

BMW has set up an experimental car with the following sensor configuration 
depicted in Fig. 1. Concentrating on the surveillance of the area in front of the 
vehicle, these cooperative sensors, which operate on the basis of distinct phys- 
ical principles, complement each other both in effective range and spatial 
accuracy. 
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Fig. 1 . BMW experimental car equipped with the following sensor config- 
uration: (a) SAGEM far infrared camera, (b) Grey-scale camera, (c) 
Bosch SGU long range radar, (d) IBEO laser scanner and (e) 
MA/COM short range radar. 

The usage of a far infrared (FIR) sensor guarantees both perception at bad 
lighting conditions and straightforward vehicle and pedestrian detection (see 
Fig. 2). As most pedestrian-scenarios covered by the experimental vehicle, are 
situated in the area to the right of the road, this sensor is mounted at the right 
of the front bumper. Long and short range radar sensors are surveying the 
environment ahead providing a seamless transition in distance and field of 
view resolution. Moreover, a laser scanning (lidar) device is mounted beneath 
the number plate to enhance the detection and tracking quality for both 
pedestrians and vehicles. The visual grey-scale cameras are used for supervis- 
ing and controlling purposes only. 




Fig. 2. (a) Pedestrian (FIR camera), (b) Pedestrian (grey-scale camera), (c) 

Vehicle (FIR camera), (d) Vehicle (grey-scale camera). 
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3 Collision Mitigation Application 

The target application of the demonstration system is collision mitigation by 
means of autonomous braking. Accident statistics tell us that most of the acci- 
dents with severely injured persons happen through collisions of cars with vul- 
nerable road users - often in urban areas on straight roads. These accidents 
can be attenuated or even prevented by our collision mitigation system. The 
second crash scenario addressed in our system deals with rear-end collisions. 

The basis for any intra-system decision is a situation assessment. Taking into 
account the geometric and kinematic data of object-models provided by the 
perception system as well as probabilistic attributes and physical limits the 
collision risk is estimated. Only in case of an inevitable collision, the system 
engages the brakes autonomously. It uses the fact that technical systems are 
capable of reacting much faster than human beings. 

Emergency braking caused by false alarms would greatly impede road safety 
improvement not lastly due to the major setback such an incident would rep- 
resent for driver acceptance. Thus such an active autonomous intervention in 
the process of driving requires an outstanding degree of perception perform- 
ance, particularly with regard to accuracy, availability and robustness. 
Therefore the attention is especially concentrated on the construction and 
design of the environment perception system. 



4 Perception System 

The central challenge for advanced driver assistance systems is an adequate 
perception of the vehicle's environment and the generation of an environment 
description out of these data. 

A multi-sensor system containing a set of sensors based on distinct physical 
principles establishes a basis for an accurate and robust environment percep- 
tion - notably if they complement each other in their sensing capabilities. Key 
ingredient of the perception system is the way how the diverse and sometimes 
conflicting measurement data from different sensors is combined in order to 
increase information content on the one hand and to reduce the amount of 
data on the other hand. 

In track-based fusion approaches several sensor data streams are processed 
independently from each other until the level of object data is reached. Fusion 
on the object level runs the risk that useful data is discarded during early pro- 
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cessing steps (e.g. data reduction or feature extraction) because it doesn't 
seem to be significant enough from one sensor's point of view. That way con- 
tradictions could arise on object level, that are difficult to be resolved due to 
the lack of lower level information. 

Therefore it is this paper's standpoint that the "early fusion concept" is the 
more promising approach to exploit the synergies of the different sensor data. 



4.1 Early Fusion Concept 

The term "early" means to combine data provided by multiple and even 
diverse sensors at an early stage of the data processing chain. These input data 
can be slightly pre-processed - limited to untracked and raw sensor data. 

During the subsequent fusion algorithm, data from one sensor is assessed with 
regard to the relevance of its information, always in the light of data provided 
by other sensors. 

The early fusion approach ensures consistency of models in the whole pro- 
cessing chain. In particular, one common environment model is used for 
describing the same aspect of reality seen from different sensors. Thus the 
whole sensor data contributes to one global environmental description, i.e. one 
outgoing result. With this early fusion approach it is expected to achieve a 
robust and reliable output of the environment perception system. 



4.2 Fusion Cycle 

Generally speaking an abstract inference problem is composed of three circu- 
lar steps namely time prediction, data association (data matching) and meas- 
urement update (correction) [4]. On top of this basic pattern we added further 
steps to come up with multi-sensor and multi-object demands. The following 
subsections describe the fundamental structure of the implemented fusion 
cycle also condensed in Fig. 3. 



4.2.1 Data Acquisition 

As most of the used sensors are working on different clock rates and time is 
crucial in collision mitigation, we preserve a high time resolution by a semi- 
asynchronous data acquisition. The fastest sensor (master sensor) with respect 
to the refresh rate is used to trigger this step. The actual data acquisition is 
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done by polling every sensor for new data. This raw data is stored in a sensor 
specific coordinate system, which is relative to the sensor's mounting position. 
Measurements of sensors, which work asynchronously to the master sensor, 
obtain a time delay as these measurements occur between the last and the cur- 
rent cycle. Assuming a maximum object velocity of 15m/s in urban environ- 
ments, this error is negligible compared to sensor specific errors. In addition, 
a slight pre-processing of the raw data, i.e. noise reduction or edge extraction, 
is performed. As the sensors have probably moved along with the own-car, a 
coordinate transformation from sensor specific to world coordinates can not be 
applied in this step. Nevertheless, this conversion is handled in the subsequent 
"time prediction" step. 




Fig. 3. Overview of Fusion Cycle. The cycle start is at the red circle. Yellow 
boxes symbolize tracked object-models. Green boxes and blue 
lines/circles represent predicted object-models respectively pre- 
dicted measurements. Red lines/circles show real measurements. 



4.2.2 Time Prediction 

According to every objects' state (position, orientation, velocity ...) at the last 
cycle, these states have to be predicted to the current time. With the nomen- 
clature from [3] the object-states x k and their prediction error covariances P k 
are projected one time step ahead via x k 



** + l = /(**>“*) 



( 1 ) 
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( 2 ) 



on the basis of the objects' underlying dynamic models f (•) and their deriva- 
tive A k 

A single track dynamic model [5] utilizing own car sensor data like current 
velocity steering angle and lateral acceleration is used to predict the own-car's 
position. As all other sensors are mounted to the own-vehicle, their position is 
directly deducible and a world coordinate transformation is performed. In 
combination with a global coordinate system, this simplifies and standardizes 
the subsequent time prediction step of tracked vehicles and pedestrians. 
Currently for these object-models a linear dynamic model is applied. 




Predicted 
Measure mente, 




Pedestrian 



Car 



O 





Own 

Car 



Fig. 4. Showcase for the predicted measurement generation. Using two 



different physical sensor principles predicted measurement gener- 
ation is illustrated for both vehicles and pedestrian object-models, 
considering the sensors' view-port as well as partial occlusions. 
(Exemplified scene is in birds-eye-view and not in scale) 



4.2.3 Predicted Measurement Generation 



In the previous step for every object an updated representation (state) with 
regard to the current time is generated. These predicted states are the basis 
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for the following step, which estimates what each sensor would measure 
under the assumption that every objects' state was correctly predicted. With 
the notation of [3], this process reflects the non-linear function h(x^) , 
where x k represents the result of the "time prediction" step. In the following, 
a representation of the general task of the "predicted measurement genera- 
tion" is given. The basic principle of this task is also shown in Fig. 4. Currently 
all object models are composed of simple polygons. 

For all valid sensors within the current cycle the following steps are per- 
formed: 

► Sensor Transformation: Transform all known object-models into the 
sensor coordinate system. 

► Back-face Culling: Temporarily remove the objects' parts which are not 
"visible" to the sensor, because they are occluded by other parts of the 
object. 

► Clipping: All objects residing outside the view-port of the sensor are 
ignored. 

► Occlusion Test: Discard all objects which are "invisible" because they 
are occluded by other objects. 

Measurement Generation: For the remaining objects respectively object parts 
the object specific predicted measurements are computed. This requires, that 
all object models have their own sensor specific representation. 

Most of the necessary steps, like sensor coordinate transformation, clipping or 
occlusion testing, are strongly related to common computer graphics tasks. 
Therefore, we use a scene-graph representation for all object-models, which 
allow for an easy adaptation of these algorithms to the specific sensor charac- 
teristics. 



4.2.4 Data Matching 

The next step within the aforementioned fusion cycle is the data association 
that extracts and assigns corresponding pairs of real and predicted measure- 
ments. Due to the large data amount compared to most track-based fusion sys- 
tems and the resulting complexity to determine the matching pairs, gating 
mechanisms are essential to support the fast finding of data correspondences. 
Thus, sensor data specific rectangular gates (derived from the object's shape 
model; see Fig. 5) 
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Fig. 5. Gate calculation of an object 

are computed from the object's predicted estimation error covariance , the 
Jacobian H k of partial derivatives of the state-to-measurement function with 
respect to states and the sensor's measurement noise covariance R k . Therefore 



8 = 




(3) 



is added to the object's length and width respectively on each side, where 
y E (0;oo) and denotes the respective element in the innovation covariance 
matrix 



S k =H k P- k H> k 



+ Rj, 



(4) 



Now the real measurement data is tested for being inside one or more gates. 
Next, the data within the gating area of an object-model is assigned sensor- 
specifically to the object's predicted measurements (as calculated in 4.2.3) on 
the basis of a Global Nearest Neighbour approach. 
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4.2.5 Hypothesis Generation 

A priority goal of the hypotheses generation is a direct and complete detection 
of all so far untracked and possibly relevant objects in the sensors' ranges. 
Thereto a high error of second kind is consciously taken into account. Usually 
a succeeding classification procedure as well as an observation of the objects 
over time can select and eliminate irrelevant assumptions. To limit the cost of 
computation, the hypothesis generation focuses to salient and unmatched sen- 
sor data in the detection range. 

Currently the unmatched salient points, where new assumptions are placed, 
are radar responses, lidar segments within certain dimensions and vertical 
image edges from the far infrared imaging device. To limit the amount of 
assumptions a first coarse pre-classification step rejects impractical assump- 
tions and a second aggregation step tries to combine overlapping hypotheses. 

4.2.6 Extended Kalman Filter 

A conventional Extended Kalman Filter (EKF) (see [1] for instance) has been 
chosen. It handles the nonlinearities of this application quite well. For every 
assigned pair of real and predicted measurement, which has been calculated 
before, a measurement update on the underlying object is performed. This 
procedure propagates the measurement information into the states of the 
respective objects. In doing so, the information of several measurements 
enhance the states by updating the objects' state values and furthermore, low- 
ering the estimation error covariances. Thereby, for each assigned sensor data 
a measurement update step is conducted before the next cycle starts with the 
object's state prediction in time. With the notation of [3] the equations at time 
step k of the EKF's measurement update can be written as 




(5) 



x k =x-+K k (y k -h(x-)) 



( 6 ) 



P k ={I-K k H k )P- k 



(7) 
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Thereby the specific term h(x j-) has already been evaluated during the calcu- 
lation process of predicted measurements and thus equation (6) can be written 
as 



x k = % + K k(yk.~ y E ) 



( 8 ) 



for every pair of (jV*/*) measurement and predicted measurement, matched 
by the data association process. As all sensor data is projected into the 3D glob- 
al world coordinate system, the entries of the Jacobian H k can be easily 
deduced from the underlying object-model without any further complex and 
time consuming calculations. 



4.3 Implementation Details 

A cyclic top-down architecture has been implemented to facilitate the detec- 
tion, classification and tracking of relevant road users over time. The real 
world vehicle surroundings and the sensor configuration are reflected by a vir- 
tual environment, which is modelled as a hierarchical scene-graph structure 
[3], ensuring centralized data access and efficient spatial dependency process- 
ing (see section 4.2.3). To allow an efficient graph traversal as well as a decou- 
pling of algorithm and data portions, the so called Visitor Design Pattern [2] 
has been used extensively. 



5 Conclusion 

This paper proposes a novel sensor fusion approach to detect and track cars 
and pedestrians in real-time to facilitate a collision mitigation application for 
vehicles. The perception system is composed of a far infrared imaging device, 
a laser scanner and several radar sensors which operate integrated into a 
BMW sedan. The proposed fusion framework in combination with the consis- 
tent use of global world coordinates for measurements, matching and the pre- 
dicted measurement generation provides a high level of abstraction. Thus, 
more sensors at different view-ports up to a 360° sensor configuration could 
easily be implemented. 



64 I Safety 



6 Further Work 

There are still ample possibilities for improving the system. The most impor- 
tant ones to be tackled in future are to extend the system by a classification 
unit, to develop an auto-calibration of sensors and to apply alternative filtering 
approaches. In addition to these improvements, an extensive evaluation of the 
system performance is planned. 
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Abstract 

The paper introduces a new driver assistance system within the 
BMW ConnectedDrive concept. Based on the driving dynamics and 
navigation data, the Dynamic Pass Prediction (DPP) indicates road 
sections that are not safe for overtaking. By reducing the enormous 
driver workload before overtaking situations a safer and more com- 
fortable driving is achieved without losing driving pleasure. This 
example shows how driver assistance systems can take advantage of 
navigation data especially if it contains curve and sign information. 
With the quality of navigation data available today the DPP function 
is feasible. Taking driving parameters into account, a situation adap- 
tive recommendation provides even more benefit for the customer. 



1 Introduction 

Imagine you are driving on a curvy road, intentionally having disregarded the 
motorway access. You enjoy the acceleration, the force in curves, proud of the 
outstanding handling of your driving machine, fading out all the strains of the 
office. Crud! A truck! Too fast to pass on the fly and too slow to follow. You pull 
left in order to get a view over the upcoming road. No, too short to the next 
curve. Again - negative again! This time you fall back and accelerate before 
you pull out, thus shortening the necessary way to pass, but the run of the 
right curve is not visible at all. Brake! You start to get ambitious - two more 
unsuccessful attempts. After the next curve you will do it . . . 

Sounds familiar? Imagine that: You have a quick look at your navigation mon- 
itor and see that passing is not a good idea for the next two kilometres; an 
extra passing lane commences after that distance. You relax and change audio 
track: " Country roads ..." 
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About 43% of severe traffic accidents today occur on rural roads. Typically 
severe accidents can be assigned to following accident sites: 33% on curved 
roads, 18% on sloped roads, 20% at or close to intersections, 29% on other 
places, see Ref. [1]. On rural roads, overtaking counts for about 10% of all 
severe accidents. Even though road traffic in Europe is steadily increasing, the 
number of fatal accidents was reduced by 40% within the last decade. This 
positive effect can be mostly attributed to automotive active and passive safe- 
ty systems such as improved braking systems, DSC, air-bags, improved car 
structures, navigation systems as well as traffic regulation, infrastructure 
design, etc. However, the number of 40,000 casualties per year in Europe is 
still too high. 

The BMW ConnectedDrive concept focuses on the intelligent integration of 
driver, car and environment concerning driver assistance and communication 
[2]. For BMW ConnectedDrive, driver assistance systems enhance safety and 
support drivers actively without interfering: they only make recommenda- 
tions. Examples of available BMW systems are Active Cruise Control (ACC) 
using navigation data. Adaptive Light Control (ALC) and BMW Night Vision 
(NiVi), see Ref. [3-5]. 

Vehicle navigation based on map data will play a major role in future BMW 
driver assistance systems. The integration of precise navigation in vehicle con- 
trol systems enables automatic adaptation of the system (its states as well as 
its parameters) depending upon the upcoming geometry of the roadway and 
memorized attributes/events using a GPS positioning system in combination 
with a digital road database. For the past years, the importance of accurate and 
up-to-date digital map information has increased dramatically. The digital 
maps are seen as a new type of sensors for the vehicles and could contribute 
to detect objects or dangerous curves beyond the horizon of the driver (and 
sensors). Driver assistance applications relying on map data need to be guar- 
anteed of a certain level of reliability and accuracy in order to provide safe and 
efficient services. 

In order to drive a vehicle well, the driver needs accurate information about 
the driving environment in addition to well-founded training, experience and 
the ability to perform routine functions. Ideally, the driver is able to estimate 
a situation fully and completely and then make the correct decisions. The driv- 
ing environment is based on one hand on the position, movement and type of 
the other vehicles on the road, and on the other hand on the route and the 
nature of the road, on traffic regulations, weather, visibility etc. 



However, because of the limitations of sensors, driver assistance functions at 
best possess only part of the information needed to describe the whole situa- 
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tion. Consequently, the driver will always experience a deficit in the expecta- 
tions if the subjectively perceived information does not correspond to the full 
picture of the driving environment. The map preview as an electronic horizon 
can act as an additional sensor that will enhance the assessment of the situa- 
tion of the vehicle. 

In this context navigation systems and their associated databases are used as 
additional forward-looking environment sensors, which make part of the miss- 
ing information available. The geometry of the road surface and other infor- 
mation about the road, such as its type, curves and the number of lanes or 
restrictions, results in an estimation of the driving environment. This gives 
rise to opportunities for optimizing the driver assistance functions. 



Fig. 1 . DPP system: red sections indicate 'not safe for overtaking'. 

The outlined assistance system Dynamic Pass Prediction (DPP, Fig.l) offers a 
new function for recommending overtaking related handling strategies. The 
fundamental algorithms of the overtaking decision taking process, the HMI 
development as well as results from real drive tests are described. 



2 Map Database 

Prototype version of DPP is based on commercial NAVTEQ Digital Maps that 
can be found in most of today's in-car navigation systems. 
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To find out of if street segment is suitable for overtaking, DPP system will 
examine number of standard street attributes such as number of lanes, form- 
of-way (motorway, single-carriage road, roundabout . . .), speed restrictions, etc. 

Another important element in realization of DPP functionality is availability of 
street curvatures. While this data is not yet available in commercial Digital 
Maps, NAVTEQ pre-calculated the curvatures at shape points using existing 
street geometry. In general, such calculation can take place on-board. 
However, to achieve better results, sophisticated spline-interpolation algorithm 
is used. This method is calculation intensive and it is not practical to perform 
it in real-time on the navigation computer. 

In the digital map database, calculated curvatures for street shape points are 
stored. In addition, on each crossing, one curvature value for every pair of 
streets is pre-calculated and stored in the database. Reverse-linear interpola- 
tion is used to emulate continuous curvature function over entire street length. 
During the road tests, it was found out that the curvature data is of good qual- 
ity. 

The legal overtaking restrictions are not part of standard NAVTEQ digital map 
attribute set. NAVTEQ collected and integrated to the database number of 
street attributes that may affect the overtaking decision process. For instance, 
positions of pedestrian crossings, traffic lights as well as street markers are 
inserted in the database. 



3 Electronic Horizon and Most-Probable-Path 

Extract of the digital map containing streets that may be reached by the car 
in the near future is called Electronic Horizon (ADAS Horizon, Extended Driver 
Horizon). Path on Electronic Horizon that will be followed with highest proba- 
bility is called Most Probable Path. Construction of the Electronic Horizon and 
Most-Probable-Path is outside of scope of this article, but it can be said that EH 
is generated using probabilistic algorithms that takes in account number of 
street attributes as well as the calculated route (if available). 

DPP algorithm examines only street segments along the Most-Probable-Path. 
While is possible that the Electronic Horizon Most-Probable-Path does not cor- 
respond with the driver's intentions, this situation is very rare in DPP. 
Typically, DPP will be used on cross-country roads where frequency of cross- 
ings is not very high. In addition, on each such crossing EH algorithm will usu- 
ally prefer most important road to follow; driver will turn to side roads either 
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when he know in advance which path he will follow (i.e. he is already aware 
of the crossing), or if calculated route follows that way. In the later case, since 
calculated route is used as parameter in EH algorithm, side road will be on 
Most-Probable-Path and the turn will be taken in account by the DPP algo- 
rithm. 



4 DPP Driving Dynamics 

Based on road geometry and other attributes in digital navigation maps as well 
as actual driving dynamic parameters DPP informs the driver about road sec- 
tions that are not safe for overtaking. In the sense of the BMW ConnectedDrive 
concept the driver gets knowledge about roads ahead even when he is driving 
in unfamiliar areas. This avoids the continuously increasing willingness of the 
driver to take an overtaking risk. By using DPP in overtaking situations a safe, 
relaxed and comfortable driving is achieved. 

Since the proposed system does not monitor oncoming traffic, it will not indi- 
cate that overtaking is safe. So the complete manoeuvre remains the driver's 
responsibility. As a matter of course the driver is obliged to obey effective traf- 
fic rules. 

In order to adapt to the actual situation DPP also makes use of dynamic vehi- 
cle information like 

► initial velocity 

► rate of acceleration 

► rate of deceleration 

The given geometrical distance values during the overtaking manoeuvres are 
shown in Fig. 2, where vehicle 1 denotes the vehicle that is overtaken and 
vehicle 2 the overtaking vehicle. 

The parameters for the overtaking distance are given by S H describing the 
whole overtaking distance, S 1 and S 2 the safety distance to the vehicle in front 
of the overtaking vehicle and after the overtaking manoeuvres, respectively. / x 
and / 2 are length of the vehicles. S L finally describes the distance that the 
overtaken vehicle has passed during the overtaking manoeuvres. The sum of 
the whole overtaking distance is than given by S 0 = S H + S L , where S H is 
given by S H = S 1 + S z + + l 2 . 



72 I Safety 




Fig. 2. Geometry parameters for the overtaking manoeuvre 
Sy depends further on 

► v 0 : the initial speed of the overtaking vehicle, which is assumed to 
be equal to the speed of the overtaken vehicle 

► v a : the assumed velocity of the overtaking vehicle after the overtaking 
manoeuvre 

► v max : speed limit 

► a a : acceleration of the overtaking vehicle and 

► a d : deceleration of the overtaking vehicle 

► as well as the range of vision and the oncoming traffic. 

Fig. 3-5 show speed plots for different situations. In Fig. 3 the overtaking vehi- 
cle accelerates to a turning speed v m and subsequently decelerates to the 
desired speed v 1 . Applying standard motion equations 

v = v 0 + a *t 



and 



S = v 0 *t + !/ 2 *a *t 2 

and eliminating variables v m , t a and t d results in the required passing way (see 
Fig. 3-5. 
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Fig. 3 



Fig. 4 






Fig. 5. Speed plot during the overtaking manoeuvres ( V m > V max ) 
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Fig. 4 shows the special case of acceleration to the desired speed without a 
deceleration phase. In Fig. 5 the maximum speed is limited by v max . 
Calculation of the passing way Sy is done analogously. The time values t a , t c , 
and f d will be calculated dynamically. 

All road sections are marked as "overtaking not recommended" except those 
with curvature attributes below a limit continuously for at least the passing 
way S y . The influence of velocity is shown in Fig. 6-8. Illustrated are map data 
on a rural road between Konigsdorf and Bad-Tolz (South of Bavaria). The 
orange road sections show sections where overtaking is not recommended for 
various velocity parameters. 






Fig. 6. - Fig. 8. 



Orange road sections indicate "overtaking not rec- 
ommend" for various velocity parameters 



5 HMI, Visualization and Results from real Test Drives 

Dynamic Pass Prediction is implemented in a 5er series sedan with navigation 
system and Head up Display (HuD). Results from real test drives tests have 
shown that the system enhances safety and supports drivers actively without 
interfering. The calculated distance is matched with map and navigation data 
where overtaking is not recommended. 

The resulting information is shown on the head up display and on the naviga- 
tion display (shown in Fig. 9 and Fig. 10). In Fig. 9 the orange colored bars indi- 
cate road sections that are not safe for overtaking and show the matched vehi- 
cle position and the electronic horizon in the navigation display. Fig. 10 shows 
the illustration on the head up display. 
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Fig. 9. Electronic Horizon indicating orange road sections in the naviga- 
tion display where overtaking is not recommended 




Fig. 10. Dynamic Pass Prediction integrated in the head up display 

If a road segment is straight and free of crossings but too short for overtaking, 
it will be marked with light orange colour. Since the overtaking distance is a 
function of speed, it is possible that road segments that are too short become 
white if the vehicle changes speed. 

Segments, which the DPP algorithm finds unsuitable for overtaking, are 
marked in orange colour (see Fig. 11). Most Probable Path (MPP) event signs 
symbolize the reason for the classification in the DPP engine. The Most 
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Probable Path (MPP) is part of the electronic horizon. If multiple reasons rec- 
ommend not passing, an MPP event sign representing the first reason will be 
shown. All the colours in visualization are freely configurable. It is possible, for 
instance, to set the colour of 'too short' segments same as colour 'do not pass' 
segments. 




Fig. 1 1 . Dynamic Pass Prediction design 

As illustrated in Fig. 11, the vehicle on the display is orange if the start of an 
overtaking manoeuvre is not recommended at the moment and white other- 
wise. 



6 Conclusion 

Driver assistance systems heavily rely on remote sensing for road traffic situ- 
ation perception. The concept of advanced vehicle navigation extends the dri- 
ver's visual horizon to an electronic horizon with a much larger range. The 
ability to combine dynamic data (such as vehicle data) and map data (such as 
the roadway geometry from a vehicle navigation system), provides the means 
for a driver assistance system to generate a "clear" picture of the current traf- 
fic scene. 
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BMW has proven by series applications, that the integration of advanced vehi- 
cle navigation in driver assistance systems can bring substantial benefits in 
terms of reliability robustness, increased functionality, fuel efficiency, active 
safety and performance. Driver assistance products using map databases as an 
additional source of information will most likely be introduced in phases. The 
Dynamic Pass Prediction is based on available GPS technologies and common 
map databases designed for driver assistance systems. Even with the quality 
of navigation data available today, BMW's DPP system can be enhanced by 
dynamics, which adapt to the situation, providing even more benefit for the 
customer. By using DPP in overtaking situations a safer, more comfortable and 
superior driving is achieved. 
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Abstract 

The possibility to assess active safety measures for pedestrian pro- 
tection is a prerequisite for the definition of reasonable driver assis- 
tance systems and their sensor need. The Verps + -index combines 
In-Depth accident data, driver models and vehicle tests in a new 
way As an example, different brake assist strategies are analysed 
regarding their safety benefit for pedestrian protection. 
Additionally, requirements for the sensor system will be defined. 
Different sensing principles can be rated regarding their compliance 
to these requirements. It can be shown that by use of environmen- 
tal sensors a brake assist strategy which keeps the driver in the loop 
already obtains remarkable safety benefits for pedestrians, while 
more advanced autonomous emergency braking strategies retrieve 
only limited additional performance. This means that mayor benefits 
in pedestrian protection can already be obtained by using "simpler" 
environmental sensors with reduced pedestrian recognition capabil- 
ities. 



1 Introduction 

A few years ago vehicle safety was all about occupant safety. The impact of 
the vehicle on its environment and to vulnerable road users (VRU) had less pri- 
ority. Huge efforts have been made in the last decades to increase road safety. 
Therefore, modern cars are unprecedently safe in terms of occupant safety. 
Nevertheless, there is still a great potential to save the lives of VRU's and to 
minimise their injuries. During the last decade pedestrian and bicyclist safety 
discussion peaked at national and international levels. Calls have been made to 
implement measures to increase pedestrian safety. This has lead to European 
legislation concerning pedestrian safety and other VRU's. The recently 
released legislation is based on a component test procedure which assesses the 
vehicle front [1]. 
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Nevertheless, active safety in general (infrastructural, educational and techni- 
cal measures) has increased, helping to keep the number of accidents nearly 
constant while traffic density is increased. Against the background of the 
development of intelligent driver assistant systems (ADAS), the European 
pedestrian safety approach has to be adapted. These active safety systems 
might be able to sense a crash before it happens and reduce its severity or 
even avoid a crash by using advanced pre-crash sensors. The following chart 
(see Fig. 1) shows an expected penetration of the car market by active safety 
systems within the next ten years in Germany. 



Assistant Systems 2003 



1. Generation (35%) 



ABS 

ESP 

Brake Assistant 
Pressure Control 
ACC 

Adaptive Light 



€ 900,- per car 



Assistant Systems 2010 




Assistant Systems 2015 




3. Generation (20%) 

+ X 

+ x 


2. Generation (30%) 


2. Generation (75%) 


Lane-Keeping-Assistant | 




Lane-Keeping-Assistant 


Lane-Change-Assistant 




Lane -Change - As sistant 


Automatic Emergency Brake 


Automatic Emergency Brake 


Wake-Up System 


Wake-Up System 


Log Sensor 




Log Sensor 


Crash Detection 




Crash Detection 


Object Detection 

Pedestrian Protection 




Object Detection 

Pedestrian Protection 


+ x 




+ x 






1. Generation (80%) 


1. Generation (100%) 






€ 3200,- per car 


€ 4300,- per car 



Fig. 1 . Market share for driver assistant systems in Germany (based on [2]) 



1 Accident Research 

Analysis of accidents involving vulnerable road users (VRU's) helps to deduce 
suitable measures of protection. Additionally, the effect in real world accident 
situations of certain measures can be proven by use of accident benefit 
research. In this chapter data related to pedestrian accident situation is sum- 
marized. The accident figures in Tab. 1 give an impression concerning the rel- 
evance of pedestrian and bicyclist accidents worldwide. 
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Region 


Pedestrians killed 


Bicyclists killed 


Pedestrian injured 


Bicyclist injured 


EU-25 


8.798 ( 2003 ) 


- 


184.300 ( 2003 ) 


- 


Germany 


8 3 8 ( 2004 ) 


475 ( 2004 ) 


33.160 ( 2004 ) 


73.000 ( 2004 ) 


USA 


4.641 ( 2004 ) 


725 ( 2004 ) 


78.000 ( 2001 ) 


41.000 ( 2003 ) 


Japan 


2.332 ( 2003 ) 


991 (2002) 


85.5 92 ( 2003 ) 


- 


Australia 


2 3 2 ( 2003 ) 


26 ( 2003 ) 


- 


- 



Tab. 1 . International comparison of accident figures concerning killed and 

injured pedestrians and bicyclists (based on current governmental 
statistics). 



Federal accident statistics are of limited use for detailed questions concerning 
the accident event or the cause of injuries. For more detailed questions, in- 
depth accident analysis can give the answers. In Germany, the "Bundesanstalt 
fur Straftenwesen (BAST)" and the "Forschungsvereinigung Automobiltechnik 
e.V. (FAT)" cooperate in conducting such studies (GIDAS ) [3]. Based on in-depth 
data of 663 pedestrian-car-accidents, occurred since 1994, detailed analysis of 
this accident type can be done. The results are limited to the suburbs of 
Hanover, slight changes are expected for accidents throughout Germany. 



1 .1 Accident Event and Location 

Analysis of statistical data shows that 94% of all pedestrian accidents happen 
in the urban environment. Only 6% of accidents happen in rural areas, but 
accidents in rural areas cause more severe and fatal injuries. 71.4% of the 
killed pedestrians were involved in accidents in rural areas. 

80.2% of all pedestrian accidents happen on roads classified as "city streets". 
9.7% occur on country roads, 4.9% on federal roads and 3.2% on parking sites. 
On other road classes, only 2 % of all pedestrian accidents take place. This is to 
differentiate between gateways (0.8%), autobahn (0.6%), cycle-, footways and 
trails (0.4%) and play streets for children (0.2%). Injury severity level depend- 
ing on road class shows that the injury severity level increases with increasing 
speed limit. Straight road is the most common accident scene in case of pedes- 
trian accidents. 60.3% of all pedestrian accidents happen in this scene. Other 
common accident scenes are crossroads with 19% of the accidents and gate- 
ways with 1 1 % of the accidents. 

82% of pedestrian accidents happen with an impact velocity up to 40 km/h. 
87.8% of accidents causing slight injuries at this velocity level. The risk of 
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more severe injuries increases with increasing impact velocity. Severe injuries 
are sustained in 61.4% of accidents at an impact velocity level up to 40 km/h. 
No deadly injuries are found in the database up to 50 km/h. 




impact speed [km/h] 

Fig. 2. Cumulative frequency of the injury severity (MAIS) depending on 
impact speed 

In car-pedestrian-accidents, the vehicle front is the most commonly impacted 
part of the car. It is hit in 70.6% of all cases. The sides of a car are often hit as 
well. Impacts to the right side (15.9%) are twice as common as impacts to the 
left side (7.3%). The rear is involved in 5% of the accidents. This shows the 
necessity to establish a test procedure that makes it possible to evaluate the 
pedestrian compatibility of a vehicle front. 



1.2 Pedestrian Data 

39% of pedestrians involved in accidents are between 18 and 64 years old. 
24% of all involved pedestrians are children between 6 and 12 years of age. 
19% of all involved pedestrian are people older than 65 years. Small children 
up to 5 years (10%) and teenagers between the age of 13 and 17 (8%) are 
rarely found in the database. A more detailed analysis of age or gender could 
show further results (see Fig. 3). Children, between ages 6 and 12, are very 
often involved in pedestrian accidents. Six year old boys have the highest risk 
to be involved in a car-pedestrian-accident (23 cases). 

In 92.2% of the cases, the pedestrian was crossing the carriageway of the vehi- 
cle before the accident happened. That is the reason why the pedestrian is hit 
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from left or right side by the vehicle. Along the vehicle front no special loca- 
tions for the impact can be found, thus an equal distribution of pedestrian 
impact points can be assumed. 



age distribution of involved pedestrians according to sex 




Fig. 3. Illustration of age distribution of involved pedestrians according to 
gender. 



Walking velocity is known either from people involved in the accident or from 
witnesses. Therefore, it is important to analyse relative movements, sight con- 
ditions, angles between witnesses and people involved. Interpretation by the 
witness is important too: Short, fast steps resemble faster movement than 
wide, slow steps. 

walking velocities 



18% 



□ other ^walking □ running 

□ in motion ■ slowly walking □ qu ickly walking 

□ standing, lying □ unknown 

Fig. 4. Illustration of walking velocities of pedestrians. 

Analysis of 609 pedestrian accidents shows that 94% of all pedestrian were 
moving directly before the impact. 152 of them were running (25%), 59.7% 
(364) were walking. Further division of this category shows that 33% (119) 
were walking quickly and 26% (96) were walking slowly. Some pedestrians 
were moving but it was impossible to characterise the movement. These cases 
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are summarized in the category called "in motion" Only 6% of the pedestrians 
were not moving just before the impact. 



1.3 Injury Frequency of Body Regions 

Injury frequency of the various body regions is shown in figure 5. The most 
injured body regions are the lower extremities (35%) and the head (33%). 
Severity of injury can vary for each body region. Head injuries can be divided 
into slight (AIS 1 + 2, 89%), severe (AIS 3-5 10%) and fatal (AIS6, 1%). 



injury frequency of body regions 



pelvis abdomen nec ^ 

5% 3% 2 % 

thorax (w/o ribs 

ribs) 2% 




6% 

upper 
extrim ity 

15% 



head 

33% 



lower leg 
13% 

knee 

9% 

lower extrimity 
not specified 
6% 

upper leg 
3% 

foot 

3% 



Fig. 5. Allocation of injuries according to body regions. 



Analysis of all vehicle parts which caused head injuries shows that impact on 
the windshield is causing 44,3% of head injuries. Impact on the windshield 
causes twice as often head injuries and severe head injuries occur three times 
often as at the bonnet. Other vehicle parts mostly cause only slight injuries. An 
exception is the A-pillar. Head impact on the A-pillar causes severe injuries in 
82% of the cases. 



1.4 Causes of Injuries at Secondary Impact 

A look at secondary impact shows that about 90% of the injuries received were 
caused by contact with road surface. A dependency of collision velocity or 
height of the pedestrian could not be proven. Injury frequency to head (37%), 
upper extremities (23%) and knee (11%) due to secondary impact is beyond 
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the average injury frequency to all body regions. Most injuries are slight non 
deadly injuries caused by secondary impact can be found in this database. 

Otte and Pohleman have analysed in-depth data material in detail [4]. They 
revealed that with increasing impact velocity more injuries to the adult pedes- 
trian are caused by primary impact than by secondary impact. At lower 
impact velocities more injuries are caused by the secondary impact (see also 
[5]). In general, primary impact causes twice as many injuries than secondary 
impact. At high impact velocities, three times as many injuries are caused by 
primary impact. Very often head and lower leg injuries are caused by impact 
with the road surface. Frequency of hand and knee injuries rises with increas- 
ing impact velocity. Head injuries caused by primary impact have the same 
level of severity as head injuries caused by secondary impact. In other body 
regions the level of severity due to primary impact is higher than the level of 
severity due to secondary impact. Injuries caused by secondary impact are not 
influenced by bonnet length and windscreen angle if the vehicle is ponton- 
shaped. Bonnet leading edge height, throw distance and collision velocity have 
major influences on injury severity due to the secondary impact. 



1.5 Conclusions of Statistical Analysis 

As shown in the paragraphs above, there are a lot of influences in a pedestri- 
an car accident. A summary of the most relevant parameters of the pedestri- 
an car accidents lead to the following conclusions: 

► 94% of pedestrian accidents happen in urban environment. 

► 70.6% of the main impact points on the vehicle are found at vehicle 
front. The sides were hit in 23.2% of the cases and 5% were hit at the 
rear of the vehicle. 

► In 80% of the cases the pedestrian was crossing the carriageway of the 
vehicle. 

► 82% of the accidents happen at an impact velocity of up to 40 km/h. 

► Frequency of damages to the windscreen, a-pillar and roof increase 
with increasing impact velocity. 

► In most cases the reason for the accident is exceeding the speed limit 
or unsuitable velocity. 

► 84% of pedestrians suffered slight injuries, 15% received severe injuries 
and 1% were killed. 

► In 58% of the cases, the pedestrian suffered head injuries. 10% of these 
were severe and 1% deadly. 

► 34% of the injured pedestrians were under 13 years of age. 
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2 Assessment of Pedestrian Compatibility with VERPS+ 

Head impact by far results in the most severe and almost all fatal injuries in a 
pedestrian-car-collision. Therefore, our approach is focused on. Numerical sim- 
ulations can provide knowledge about the kinematics of the event for each par- 
ticular car and for diverse impact conditions. The results of the simulation are 
used to control a free flying head-form test device. The measured values pro- 
vide the basis for the assessment of each analysed car. 

Our assessment procedure therefore combines the following four modules: 

► accident analysis 

► numerical simulation 

► component test 

► quantification of pedestrian safety 

The interfaces between the modules are well defined in a way that our "plug- 
gable" method can take in new or other data in order to adapt the VERPS- 
Index on latest research results or new injury assessment approaches. In order 
to quantify pedestrian safety and to make sure that the results are compara- 
ble for all forms of vehicles on a linear scale, a Vehicle Related Pedestrian 
Safety Index is proposed (VERPS-index). In addition, it provides the opportuni- 
ty to assess technical measures applied to the car's front to increase pedes- 
trian safety and allows to compare them with active safety measures applied 
to the vehicle (e.g. brake assist system). 

Furthermore, the presented method makes it possible to influence the pedes- 
trian compatibility of a product in a very early stage of the vehicle develop- 
ment process by making geometry changes with minor stylistic or functional 
effects or by triggering the development of additional pedestrian protection 
systems. 

The results of the component tests can be represented by HIC values (Head 
Injury Criterion), calculated from the measured head form accele-rations. They 
show a typical pattern of potentially dangerous regions at the vehicle front: 

► parts of the bonnet with little deformation space beneath 

► lateral bonnet edge and transition area between bonnet and wing 

► bonnet area directly above the firewall 

► lower windscreen frame 

► A-pillars 

► upper windscreen frame and roof frontal edge 



All of these areas are characterized by stiff and hence less deformable vehicle 
structures. The degree of exposure of a pedestrian to these regions can differ 
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from car to car because of differences in dimensions and styling. A test pro- 
cedure which stringently demands meeting specific test limits will unavoid- 
ably produce meaningless ratings in these areas. Because of differing vehicle 
geometries the potentially dangerous areas for the head impact are affected 
more or less frequently during a pedestrian impact. Some of these areas might 
be totally irrelevant for the head impact of a pedestrian (e.g. upper wind- 
screen frame at SUV's). That's why a weighting of the test results concerning 
their relevance in pedestrian accidents is necessary. 



VERPS -methodology in order to assess 
vehicle related pedestrian safety 




Final results : 

• assessment on a linear scale 

• assessment divided by risk groups 

• assessment of technical measures 



Fig. 6. Illustration of the methodology to assess vehicle related pedestrian 
safety. 



In the following a Vehicle Related Pedestrian Safety Index (VERPS-index) is 
developed. This index assesses the level of protection which a specific vehicle 
can provide for the head of a pedestrian who is impacted by the front of the 
car. It allows to assess differences between particular vehicle designs and to 
compare technical measures applied to the vehicle front. The VERPS-index is 
the result of the quantification module in the proposed methodology (Fig. 6). 
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2.1 Mathematical Derivation of the VERPS-Index 

The VERPS-index for the frontal impact is deduced in three steps from the val- 
ues My measured in the component test: 

Mapping of M y to the degree of performance Ey by an evaluation function 



E,= B(M , ) 



( 1 ) 



Weighting of the degrees of performances Ey with relevance factors R iWAD and 
Rj Fronts deduced from accident analysis. 



Ri,WAD * -R-j, front' Eij 



( 2 ) 



Summation of the degrees of performance for all subareas of the vehicle 
front to the VERPS-index. 



VERPS — X X R,.wAn • Rj, from’ Eu 



(3) 



To assess the vehicle front it is necessary to divide it in subareas. For each of 
them My is measured. Subindex / describes the longitudinal direction of the 
vehicle front and subindex j the transverse one. 

The definition of limit values is often used to assess measured values. But it 
allows only a binary assessment. It only distinguishes between good (limit 
met) and bad (limit exceeded). In order to derive a more refined evaluation an 
assessment function B can be used to get a functional link between measured 
values M and the degree of performance E. For the VERPS-index a functional 
relationship between HIC data and the occurrence of severe head injuries (AIS 
3 + ) is used. In future, better injury risk assessment functions can easily be 
implemented. 

The reduction of a HIC-value by 50%, e.g. from HIC = 4.000 to HIC = 2.000, 
improves the degree of performance E only from E(HIC = 4000) « 1 to 
E(HIC = 2000) = 0,94 (Fig. 8). In contrast to that an improvement from HIC = 
2.000 to HIC = 1.000 leads to a significant improvement to E(HIC = 1000) = 
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0,24; this means a probability for the occurrence of severe head injuries of 



0 250 500 750 1000 1250 1500 1750 2000 2250 2500 2750 3000 

HIC 

Fig. 7. Correlation between measured HIC data and probability of AIS 3 + 
injuries (ISP - Injury Severity Probability, based on [6]). 

In the second step of the calculation of the VERPS-index the degrees of per- 
formance Ejj are weigh-ted with their relevance in accident events. The impor- 
tance of a test point obviously depends on the probability of hitting it in real 
life. In order to deduce the relevance factors, "in-depth" accident data of the 
Medical University of Hanover are used. The relevance factor in the longitudi- 
nal direction of the vehicle (R { WAD ) describes the correlation between the vehi- 
cle specific kinematics factor f K and the size of the pedestrian. In the trans- 
verse direction of the vehicle front an equal distribution for impact locations is 
assumed. This is supported by accident data. 

We calculate the VERPS-index separately for children younger than 12 years 
and for adults and children older than 12 years. However, other separations 
are possible. Our choice considers the different requirements of pedestrian 
safety measures applied to cars for children and adults which result from dif- 
ferent body heights. By use of the assessment function B (M^) the VERPS-index 
can be expressed as follows: 



24%. 



Injury severity probability for AIS3+ head injuries 




100% 




( 4 ) 
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Fig. 8 shows the division of the vehicle front in subareas and their relevance 
factors /? i#WAD for a sample car. 




Fig. 8. Relevance factors in longitudinal direction WAD ) for car F. 

The VERPS-index can run between (nearly) zero (no risk for AIS 3+ head 
injuries) and 1 (maximum risk for AIS 3 + head injuries). A car, which has a 
HIC-value of 1.000 in all subareas of its front, would have a VERPS-index of 
VERPS = 0.24. It can be shown that the proposed procedure allows to assess 
vehicle fronts on a linear scale within the limits of accuracy of the assump- 
tions. 



2.2 Application of the VERPS-index on two Vehicles Passive Safety Measures 

The VERPS-index is calculated for two sample cars. It can clearly be seen, that 
pedestrian safety has to be assessed separately for children and adults. 
Pedestrians hit different areas at the vehicle front because of their different 
body heights. This is the reason why a particular technical measure can posi- 
tively affect all groups of pedestrians only in exceptional cases. 

Two mass-production vehicles are compared with two modification levels of a 
possible pedestrian protection system. The first level represents a mechanical 
system which uplifts the bonnet in the rear area by around 0.1 m in case of a 
pedestrian impact. In the second level an airbag system is assumed which 
combines level one measures with an energy absorbing device covering criti- 
cal areas of the A-pillars and the lower windscreen frame (see Fig. 9). Results 
can be seen in Tab. 2. 
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Fig. 9. Implementation of a system to uplift the bonnet by use of an airbag 
which also covers the A-pillars and the lower windscreen frame. [7] 







vehicle F 
(mid-size) 


vehicle G 
(compact) 


production condition 


children 


0.54 


0.63 




adults 


0.63 


0.24 


uplifting bonnet 


children 


0.22 


0.43 




adults 


0.60 


0.24 


uplifting bonnet combined with an airbag 


children 


0.08 


0.11 




adults 


0.25 


0.17 



Tab. 2. Assessment of two different cars and pedestrian protection sys- 
tems by use of the VERPS-index. 



For vehicle F the VERPS-index for adults could be reduced from 0.63 to 0.25, 
for children even from 0.54 to 0.08. The marked reduction of VERPS for chil- 
dren shows the great potential of active structural measures, if they are 
applied properly with respect to pedestrian body height and the vehicle 
dimensions. Head impact areas, which are mainly hit by adults, can only be 
protected with the uplifting bonnet and the additional airbag to cover A-pillars 
and lower windscreen frame taking into account. 
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The VERPS-index of 0.24 for adults of vehicle G in production condition is good 
compared to vehicle F. This can be traced back to the fact, that all relevant 
head impact areas for adults are in the windscreen area, which is considered 
uncritically concerning the HIC results unless the windscreen frame area or 
the A-pillars are included. 

Because of the vehicle front geometry of car G an uplifting bonnet alone can 
protect only a small group of pedestrians. An additional airbag applied to the 
lower windscreen frame is able to improve the protection of smaller adults, but 
the relevant impact areas for larger ones are still not covered. Accordingly the 
VERPS-index is only reduced from 0.24 to 0.17. For children vehicle G in pro- 
duction condition performs poorer than for adults with an VERPS-index of 
0.63, because children mainly hit the firewall and the lower windscreen frame 
with the head. By use of active structural measures the VERPS-performance 
can be clearly improved. The VERPS-index decreases from 0.63 in production 
condition to 0,43 for the uplifting bonnet alone and to 0.11 for the uplifting 
bonnet with the additional airbag. 

It could be shown that an index can be formulated that allows to assess dif- 
ferent vehicles with respect to their pedestrian safety on a linear scale. The 
VERPS-index allows to compare different vehicles and applied technical meas- 
ures like the uplifting hood concerning their pedestrian protection potential. 
The injury risk induced by the secondary impact on the road is currently not 
assessed within VERPS. 

The assessment of active safety measures (e.g. pre crash sensing devices, 
brake assist systems) on the VERPS-scale is also possible. The reduction of the 
collision speed of a vehicle, which can be attained with a certain probability 
depending on the system layout, can be included in the VERPS-calculation. 
The reduced collision speeds are used as an input parameter for the numerical 
simulation module. This finally results in specific VERPS-index for the analysed 
car. 



2.3 The VERPS+- Index to assess Safety Benefits of Collision Speed 
Reduction 

The reduction of collision speed results in a shift of the relevant head impact 
regions on the vehicle front and, leads to a reduction of the induced head 
accelerations, depending on the underlying vehicle structures. Four mayor 
effects can be noticed: 

► On very stiff structures (e.g. A-pillars), the accelerations are reduced, 
but remain critical. 
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► "Critical" structures like the fenders induce lower accelerations, but still 
need structural improvements. 

► Contact with stiff areas beneath the front hood can be avoided at a cer- 
tain impact speed. This lowers the accelerations to a remarkable extent. 

► "Uncritical" areas induce even lower accelerations. 

Analysis of in-Depth accident data shows, that the head impact probability 
decreases with decreasing collision speed. This effect is taken into account in 
the VERPS + -Index by multiplication with the head-impact-probability factor 
P impact( v k) ( Fi 9- 10 and equation 5). 




0 20 40 60 80 100 120 

collision speed [km/h] 

Fig. 10. Head impact probability derived from analysis of In-Depth accident 
data 

The positive effect of collision speed reduction on pedestrian safety has been 
proven in a general approach in a couple of investigations related with the 
introduction of phase II of the pedestrian legislation starting from 2010. A ret- 
rospective estimation of the safety benefit does not take improvements of the 
vehicle structure into account, and is therefore always based on vehicle fleets, 
which are not optimised regarding passive pedestrian protection measures. 
Along with the new type approval requirements, some benefit in pedestrian 
protection can already be obtained, that could lower the additional benefit of 
brake assist systems (Fig. 1 1). 

By replacing the component test within VERPS by a numerical one (Fig. 6), the 
injury risk estimation of multiple collision speeds becomes possible. On an 
exemplary vehicle, nine impact points have been virtually analysed for colli- 
sion speeds from 20 km/h up to 50 km/h. The virtual test conditions for the 
analysed collisions speeds are calculated within the kinematic analysis of the 
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VERPS-procedure. As a result, a VERPS + -index for each analysed collision 
speed can be given (see equation 5). They can be interpolated to obtain a col- 
lision speed dependent head injury risk for the analysed vehicle. 



VERPS + k = P Impact (v)-^R iWAD (v) • 

M 



f 9 

XI- 



f HICy (>590 



19 



(5) 



The VERSP + - Index can assess the mayor injury risk at different collision 
speeds vehicle-specifically. When applicated on the exemplary vehicle, a 
reduction of the head injury risk can be derived from the virtual component 
tests (Fig. 11). 
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Fig. 11a. Vehicle specific impact of collision speed on the VERPS + -index for 
the analysed vehicle H (adults). 

For children, a remarkable reduction of injury risk can be obtained. Starting 
from VERPS + (50 km/h) = 0.53, a linear decrease of injury risk with lower col- 
lision speed is noticeable. This is mostly due to the head impact areas on the 
vehicles front hood. In this area, the head accelerations can be lowered to 
acceptable values by reduction of collisions speeds. In contrast to the injury 
risk for children, the injury risk for adults does not show a reduction over all 
collision speeds. At lower speeds, many adults have a head impact on the lower 
windscreen frame, which is a very stiff area. Therefore, reduction of collision 
speed alone is not sufficient to protect adult pedestrian in real life. But a reduc- 
tion of collision speed in combination with basic structural measures shows a 
very good performance. 



□ production model 

□ tod: 2/3 HIC<1000 

□ hood: 3/3 HICdGOO 

□ opt hood + lower windscreen HfC<200Q 
■ opt hood + lower windscreen airbag 
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Additionally, one has to keep in mind, that the derived reductions of head 
injury risk is strongly vehicle specific. Other vehicle front geometries with dif- 
ferent stiffness patterns will lead to different impact kinematics and head 
injury risks. 




Fig. lib. Vehicle specific impact of collision speed on the VERPS + -index for 
the analysed vehicle H (children). 



3 Collision Speed Reduction of Different Brake Assist Strategies 

The introduction of driver assistance systems has changed the understanding 
of an accident. In parallel to the upcoming of integrated safety approaches, 
the view of an accident has been widened onto the phase directly before an 
imminent crash. Up to now, only passive safety systems are assessed in type 
approval test and consumer tests. Nevertheless, active safety features like ESP 
offer a proven benefit for road safety [8]. For brake assist functions, a similar 
benefit is expected. A specific safety benefit of driver assistance systems could 
only be proven in retrospect accident analysis [9]. With VERPS + , the safety 
benefit of a reduced collision speed can be quantified prospectively and specif- 
ically for every analysed vehicle. Additionally, a comparison with passive safe- 
ty measures regarding their safety benefit in real life accidents is possible. 

Based on the drivers options to avoid a pedestrian accident, the precrash-phase 
can be divided into three phases (v^40 km/h): 

0. a collision warning is possible and reasonable (> 16 m, depending on the 
drivers reaction time f reaction > 1 s) 

1 . collision can be avoided by braking only ( ~ 7 m) 
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2. collision can be avoided by steering only (~6m) 

3. collision can be avoided by combined steering and braking only 5 m) 




Fig. 12. Latest possible obstacle avoidance times (TTC), when only steering 
to the left is allowed (middle-class sedan). 

The given distances have been derived from vehicle handling test with a mid- 
dle-class sedan (m= 1800 kg) and an obstacle in the middle of the road strip. In 
a speed range from 30-50 km/h (mostly all pedestrian accidents happen up to 
this speed), an upcoming collision can be avoided by the driver up to a very 
short time period before the accident. Thus, autonomous braking systems, 
which are activated only during the last unavoidable time period, are limited 
regarding their safety benefit [10]. 




Av, 



VERPS+ 



Fig. 13. Scheme of automated case-by-case-analysis of pedestrian accidents 
in order to assess potential reduction of collision speed using vehi- 
cle specific injury risks (Fig. 11). 



In 64% of all analysed pedestrian accidents (data source MH Hannover), the 
driver performed a braking before the accident. The lifting of the throttle 
pedal during the drivers reaction can be used by future brake assistant sys- 
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terns (BAS) to initiate an optimised braking. Analysis of conventional BAS in 
driving simulations showed, that in pedestrian accidents only 50% of braking 
drivers could activate the brake assist system [11]. 

Based on the fact, that future driver assistance systems should only support 
the driver in performing the driving task and therefore, the driver has always 
to be in control of the vehicle, different brake assist strategies are defined, that 
fulfil these requirements. They are assessed by their potential reduction of col- 
lision speed. This is done by an automated case-by-case-analysis of each acci- 
dent. 



Simulated emergeny braking - moving and brake response time 




Deceleration histories of TUB Drivers 




Fig. 14. Characterisation of TUB Drivers by moving time from throttle to 
brake pedal and brake pressure histories 

The recorded In-Depth accident data (initial speed, collision speed, road fric- 
tion) are used as input for a driver model, which currently consists of a set of 
54 drivers performing an emergency brake (reaction times, moving times, 
brake pressure histories). The set of drivers used in this article is called TUB 
Drivers. Other sets of drivers can be implemented too, thus leading to differ- 
ently reduced collision speeds. To determine moving times and brake pressure 
history from a realistic test, the subjects were instructed to drive on a free 
straight road with approx. 50 km/h. To the flash of a small lamp they should 
react with a full brake manoeuvre. The measured data was used to determine 
reaction times, moving times, brake pressure build-up time und histories of 
brake pressure and deceleration (Fig. 14). Wide dispersions of the histories are 
noticeable, which may be due to differences in the brake force. By using a 
brake assist system this time window can be minimised. The moving times 
show in contrast a low dispersion (Fig. 14). With the set of drivers that results 
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from the test it is possible to virtually reconstruct accidents which happened 
in reality and to predict the benefit, which can be obtained from different 
brake assist strategies. 

The driver input controls a nonlinear single track vehicle model (open loop). 
For each calculated collision speed, a VERPS + -value is calculated, representing 
the induced head injury risk. By averaging all calculated VERSP + -Values, a 
safety benefit of the analysed brake assist system in real-life accident situa- 
tions can be derived. Both the driver model and the vehicle model are derived 
and validated from dedicated tests with 98 subjects driving a fully instru- 
mented mid-size sedan [12,13]. 



In this article, three different brake assist strategies are analysed: 



► standard BAS: 



► advanced BAS I: 



► advanced BAS II: 



full brake, when certain levels of pedal 
velocities are exceeded 
full brake, when throttle pedal is lifted and 
an obstacle in front of the car is detected 
within a critical range 

partial brake with 0.3 g, when driving situa 
tion is critical (ttc < 1 s to an detected obsta 
cle) and full brake, when throttle is pedal is 
lifted and an obstacle is detected within a 
critical range 





Brake Assist strategy: 




Operating state probabilities 




Real-life 

activation 


Driver is 


' braking 


Driver is not braking 




when: 


probability 


BAS 

active 


BAS 

inactive 


BAS active 


BAS 

inactive 


Standard-BAS 


certain levels of pedal speeds and 
pressure rise time are exceeded 


50% [11] 


0,32 


0,32 


0 


0,36 


aBASI 

with environmental 
sensors 


Throttle is lifted and an obstacle 
is detected 


85 % (est.) 


0,54 


0,1 


0 


0,36 


aBASII 

with environmental 
sensors 


Throttle is lifted and relevant 
obstacle is detected 

Partial brake in critical driving 
situations (ttc<ls) 


95 % (est.) 


0,61 


0,03 


Autonomous 

action! 

0,34 


0,02 



Tab. 3. Analysed brake assist strategies and depending operation state 
probabilities. 



All relevant operating state probabilities have to be defined and their depend- 
ing lowered collision speeds have to be calculated. There are four possible com- 
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binations of driver is braking or not, brake assist activation or not, leading to 
four operating state probabilities. These reduced collision speeds are used to 
assess the safety benefit of the analysed brake assist system. 

According to the previously defined brake assist strategies, a conventional BAS 
only affects 32% of all pedestrian accidents and aBAS I can affect 54% of all 
pedestrian accidents. The only way to cover the large number of unbraked 
accidents is to start an autonomous action. This is done by aBAS II in 32% of 
all pedestrian accidents. The advanced brake assist system I, which performs 
a full brake, when a critical obstacle is detected and the driver lifts the throt- 
tle, can start an optimised braking (at around 1 g) 0.3 s earlier (average over 
all analysed drivers), thus saving 1 1 km/h collision speed (median value of all 
braked accidents) and avoiding more than 22% of all pedestrian accidents. 
Nevertheless, the positive effect of aBAS I depends on the initiation of the 
drivers brake reaction. On the other hand, aBAS I does not need any pedestri- 
an classification capability, but has to recognize pedestrians as obstacles up to 
a range of around 20 m (Tab. 5). 
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Fig. 15. 
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exemplary boxplot of calculated reductions of collision speed for 
aBAS II [13], when TUB Drivers are applied on MHH data . 



Additionally to aBAS I, the aBAS II systems performs a partial brake in critical 
situations and can reduce collision speed of unbraked accidents of 11 km/h. 
When the driver is braking, the collision speed can be reduced of 15.9 km/h 
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(median value of all braked accidents). More than 32.2% of all accidents (most- 
ly those with lower collision speeds) can be avoided by the use of aBAS II. It 
has to be indiced, that only a light partial braking is performed autonomously. 
It depends on the specific layout of such a driver assistance system to min- 
imise misuse cases and ensure driving pleasure. 





Percentage of 
avoided 
accidents 
(TUB drivers) 


median Av k when 
driver is braking 

(TUB drivers) 


median Av k 
when driver is 
not braking 
(TUB drivers) 


conventional BAS 


6.0 % 


2,6 km/h 


0 km/h 


aBAS I with 
environmental sensors 


22.3 % 


11,5 km/h 


Okm/h 


aBAS II with 
environmental sensors 


32.2 % 


15,9km/h 


Autonomous 

braking! 

1 1 ,6km/h 



Tab. 4. Analysed brake assist strategies and possible reduction of collision 
speed, when TUB Drivers are applied on MHH data. 



By using the VERPS + -index to assess the safety benefit of the analysed driver 
assistance systems, a comparison of active and passive safety measures 
becomes possible. All calculated VERPS + - values are summarized and weight- 
ed taking the previously defined operating state probabilities in real life acci- 
dents into account. 






■ no BAS 

□ ranv. BAS 

□ aBASl 
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production model hood: 2/3 hood: 3/1 optimised hood + optimised hood + optimised hood + 

HIC<1000 HlCclOtiO lower windscreen lower windscreen windscreen and A- 

slrbag HKX2Q00 pillar airbag 



Fig. 16. Average risk for severe head injuries for adults depending on 
implemented structural and active safety measures applied on 
exemplary vehicle H (TUB Drivers and MHH data). 
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4 Deriving Sensor Specification from Accident Data 

Non-contact sensors detect a pedestrian prior to the first contact with the vehi- 
cle. The use of such sensors is therefore helpful to activate pedestrian protec- 
tion measures in advance. Detection of a pedestrian prior to an accident can be 
based on different sensor principles. Of course all these sensor principles have 
advantages and disadvantages, but at least a pedestrian detection up to a cer- 
tain range has to be possible. The sensor systems are assessed for their ability 
to trigger the analysed advanced brake assist systems. Therefore, a unique 
pedestrian classification performance is not necessary. All future environmen- 
tal sensor systems must work under all possible weather conditions, offering a 
detection performance which is at least equal to the human eye. Optic sensors 
like lidar or computer vision system have disadvantages in this areas, but dur- 
ing the last years, these disadvantages have been minimized to a negligible 
amount. 
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. 17. Average risk for severe head injuries for children depending on 
implemented structural and active safety measures applied on 
exemplary vehicle H (TUB Drivers and MHH data). 



4.1 Pedestrian Detection Performance 

Depending on the type of brake assist system, a certain level of obstacle detec- 
tion performance must be reached. It is assumed, that the first generation of 
environmental sensors cannot detect more pedestrians than 85% in front of 
the vehicle. A high false alarm rate cannot be obviated, thus making any 
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autonomous braking manoeuvre impossible. For autonomous partial brake 
manoeuvres in critical situations, the pedestrian detection performance should 
be as high as 95%, thus a false alarm rate near zero becomes possible. The 
near-zero false alarm rate makes light autonomous braking manoeuvres possi- 
ble and reasonable. 
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Fig. 18. Calculated distance to the pedestrian, when the drives starts to lift 
the throttle (virtually reconstructed from MHH data, n = 254, using 
TUB Drivers, see Fig. 13) 



4.2 Range and Field of View 

Non-contact sensors are intended to detect a pedestrian over a certain dis- 
tance. Due to the lower speed range of pedestrian accidents, range and field of 
view of pedestrian detection sensors are different from those used in ACC 
applications. Detailed analysis of pedestrian accident scenarios (MH Hanover 
data) shows, that the distance to the pedestrian is below 20 m in 90% of all 
cases, when the driver starts lifting the throttle. At this time, the driver assis- 
tance system has to "know" about the driving scenario, thus a range of 20 - 
25 m would be adequate to implement pedestrian protection features. When 
talking about the sensors field of view, one has to keep in mind, that over 90% 
of all pedestrian accidents happen in urban environment, nearly 48% of urban 
pedestrian accidents happen at crossroads (VTIV data). Therefore, the envi- 
ronmental sensor system needs a much wider field of view than current ACC 
sensors. With the introduction of ACC Stop&Go functionality, a wider field of 
view is needed too. 
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4.3 Determination of Position, Sample Rate and Tracking Ability 

The needed accuracy of position determination is crucial for system function- 
ality The driver assistance system has to decide whether a driving situation is 
critical or not. A clear view of the vehicles environment is needed. Detailed 
sensor specifications on position accuracy and sample rate must be estimated 
and verified in testing procedures. Scanning lidar systems fulfil all require- 
ments regarding positioning accuracy very good, due to its high angular reso- 
lution. Radar signals are more cluttered and with inferior lateral and height 
resolution. The sample rate of Lidar and Radar systems is quite good already. 

The positioning accuracy of computer vision systems is very much dependent 
on the resolution of the camera and the system setup. Monocular cameras 
need special environment models to gain depth information. Stereovision per- 
forms better in analysing 3D environmental information. A higher amount of 
data and therefore the more complexity of data analysis results in lower sam- 
ple rates. Future pedestrian detection algorithms will run highly optimised on 
special ECU's. 

The ability to track a pedestrian over a number of signal samples is helpful to 
understand the traffic situation, thus the false alarm rate is reduced and sys- 
tem activation probability in real-life accidents is increased. It is highly 
dependent on the quality of the received data and the analysing software algo- 
rithm. In some cases, neural networks are used for object tracking. As it can 
be seen in literature, the best performer in pedestrian tracking is the ALASCA 
laser scanner [10], followed by computer vision systems [14]. 



4.4 Pedestrian Classification 

Many investigations and research work has been done in the last years in 
order to detect pedestrians in a traffic scenario and discriminate them from 
other relevant objects. Obviously, the desired driver assistance systems has 
huge influences on the demanded pedestrian classification performance. In 
this paper, two advanced brake assistance strategies have been presented, 
which do not need a pedestrian classification or discrimination from other 
objects at all. Keeping the driver in the loop means in that case, that the pedes- 
trian classification task is not performed by the assistance system, instead it is 
delegated to the driver. By evaluating the drivers reaction on a detected obsta- 
cle, a check of the obstacles relevance on the current driving task becomes pos- 
sible in a quite simple way. On the other hand, false activations of the driver 
assistance system are minimized. 
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Measurement 

category 


Brake 

assist 

system 


required 

values 

(Expert's 

opinion) 


Computer vision 

Far Visible 

Infrared and NIR 


Radar 


scanning 

Lidar 


Range 


aBAS I and 
aBAS II 


20 m (90 %) 


+ 


+ 


++ 


++ 


Object 

tracking 


aBAS I and 
aBAS II 


helpful 


+ 


+ 


O 


++ 


Determination 
of position 


aBAS I and 
aBAS II 


±0,1 m 


++ 


++ 


+ 


++ 


Sample rate 


aBAS I and 
aBAS II 


min. 10 Hz 


+ 


+ 


++ 


++ 


Object 


aBAS I 


85% 


+ 


+ 


0 


+ 


detection 


aBAS II 


95% 


O 


0 


0 


O 



Tab. 5. Ranking of possible object detection technologies for active safety 
pedestrian protection systems 

Due to the lack of a sensor that fulfils all requirements for pedestrian detec- 
tion, the use of multiple sensors seems to be the way to go. A combination of 
a computer vision system (active NIR and visible light) and a laser scanner 
would be most promising. First results show that both techniques are able to 
detect pedestrians with appropriate performance (redundancy to decrease 
false recognition rate) but are complementary enough to each other to 
increase the overall pedestrian recognition performance due to sensor fusion 
effects (both sensors use data from the opposite sensor for pedestrian classifi- 
cation). Nevertheless, radar technologies may have advantages in vehicle pack- 
age and life-time durability. 



5 Summary 

The VERPS-method to assess the pedestrian compatiblity of a vehicle front has 
been presented. An evaluation of the influence of collision speed on the risk 
for severe head injuries has been done by expanding the VERPS-Index to 
VERPS + . The exact safety benefit of brake assist systems depends on vehicle 
specific properties, driving behaviour and accident scenarios. A group of driv- 
ers has been evaluated in order to verify the wide range of possible human 
interaction in emergency brake situations. It has been shown, that the addi- 
tional introduction of brake assistance systems is equivalent to the protection 
potential of additional structural measures (active or passive). Nevertheless, 
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both types of pedestrian protection measures complement one another nearly 
ideally It has been shown, that fully autonomous emergency braking 
manoeuvres are limited regarding their pedestrian protection potential. 
Keeping the driver in the loop and adapt driver assistance strategies in a way 
that the driver always keeps responsibility on the driving manoeuvres ensures 
customer acceptance and product liability 
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Abstract 

The enhancement of comfort and safety is of major importance in 
today's research on automotive applications. Since visual informa- 
tion and image processing can significantly improve driver assis- 
tance systems, cameras are increasingly becoming an integral part 
of these systems. This paper presents an inertially stabilized camera 
platform for target tracking. The key point is the use of biomimetic 
algorithms for adaptive compensation of manufacturing tolerances. 
Therefore low-cost components can be used which make the system 
affordable in a wide range of vehicles. Initial road trials have shown 
the clear advantage of a stabilized camera over conventional car- 
fixed settings and demonstrated the usability of the proposed algo- 
rithms. 



1 Introduction 

Competitive driver-assistance systems are relying more and more on camera- 
vision systems. Wide angle cameras in the visible or infrared (IR) range are 
generally used to provide an overview of the environment and to detect road 
markers, cars and obstacles. However, a closer look at points of interest like 
number plates, traffic signs or children playing on the sidewalk requires cam- 
eras with a telephoto lens. These cameras have focal lengths of 16 to more 
than 25 mm and aperture angles of less than 10°; they are thus extremely 
sensitive to motion, especially to rotation. Even a small bump can induce 
1007s rotation rates and cause the target on the camera chip to be lost. This 
makes active camera stabilization necessary. Since a camera operates at the 
usual frequency of about 25 Hz and a latency of 30 ms due to image process- 
ing, inertial camera/gaze stabilization with inertial measurement units (IMU) 
is the only way to ensure stable images during motion in the 2-15 Hz region. 
One of the goals of FORBIAS, a research cooperation of universities, car man- 
ufacturers and major automotive suppliers for biologically inspired driver 
assistance systems, is to achieve robust active camera stabilization with afford- 
able and small hardware. The disadvantage in accuracy of relatively cheap 
MEMS sensors is compensated for by multi-sensor fusion of inertial and visual 
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information, and by using adaptive algorithms inspired by the latest develop- 
ments in neuroscience. 



2 Biological Analogy 

Biological systems, from the earliest vertebrates to humans, have fulfilled the 
task of gaze stabilization by means of the vestibulo-ocular reflex (VOR), since a 
stabilization via visual signals would be too slow because of the long process- 
ing delays (™100 ms) of the visual system. The vestibular organ in the inner 
ear is the biological equivalent of a 6-DOF IMU and measures the motion of the 
head in space. The raw sensor signals are transmitted along a short-latency 
(7 ms) direct pathway through the brainstem to the extra-ocular eye muscles 
to stabilize gaze. A second adaptive path [1] utilizes the same vestibular infor- 
mation but distributes it onto the so-called parallel fibres, bundles of nerve 
fibres extending in the cerebellar cortex. Additionally, parallel fibres also carry 
visual information and internal feedback signals related to the current eye 
velocity. The parallel fibres form a multitude of adaptive synapses with the 
cerebellar Purkinje cells. These cells, gathering the synaptically weighted 
information transmitted by the parallel fibres, produce the output of the cere- 
bellum, which is fused with the direct pathway in the brainstem. The adaptive 
process itself depends on the action of climbing fibres, another source of input 
to the cerebellum. Climbing fibres are driven by visual information about sta- 
bilization errors and act on a one-to-one basis on the Purkinje cells. By enhanc- 
ing inertial with visual information, the biological system achieves excellent 
results in gaze stabilisation by adaptive calibration of the inertial sensors. 



3 General System Description 

This section gives a general overview of the whole system and detailed infor- 
mation about main subsystems. An IMU senses the vehicle motion in space 
and is the equivalent of the vestibular organ in the human inner ear. Its digi- 
tized output is sent via CAN bus to the main DSP (dSPACE DS1103 PowerPC) 
where all biomimetic algorithms for gaze stabilization are implemented. The 
DSP output, the required angles for each axis of the camera motion device, are 
transmitted to Maxon servo amplifiers (ADS). Motor positions measured by 
incremental encoders are fed back to the control algorithm. The camera 
motion device itself consists of two or three actuated axes in gimbal configu- 
ration. 
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Fig 1 . System setup for inertial camera stabilization 



3.1 A highly miniaturized inertial Measurement Unit 

The market for inertial measurement units is dominated by few global players, 
many of them being value-adding resellers of individual sensors from compa- 
nies like Analog Devices Inc. Even the smallest IMUs currently available, like 
the Cloud Cap Technology's Christa IMU with dimensions of 50x38x25 mm 3 
including analog-digital converter (ADC) and DSP, are far too big for the use 
with driver assistance systems where the IMU should more or less vanish 
behind or even in the rear-view mirror. 

Therefore, a suitable inertial measurement unit was developed within the 
FORBIAS research project. In cooperation with the project's industrial auto- 
motive partners, main requirements for the new device were specified: 

► Design envelope of less than 20 mm in each direction 

► Mass less than 1 5 g 

► Measurement range of up to 3007s and 2 g 

► 12 bit resolution 

► Bandwidth up to 100 Hz 

► Frame rate up to 1 kHz 

► CAN and RS232 Interfaces 

Components 

After intensive tests, the newly released LIS3LV02DQ sensor 
(STMicroelectronics) was chosen as a component for the IMU. This highly inte- 
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grated 3-axis accelerometer has a dynamically settable measurement range of 
2g or 6g. The key advantage of this device is its small system size due to the 
integration of three axes on one chip and the direct digital output of the chip. 
Thus, it can be directly connected to a microcontroller using an SPI Interface. 
As angular rate sensor, the ADXRS300 (Analog Devices Inc.) has been chosen. 
Since it is pin-compatible to the ADXRS150, the same design can be used for 
applications which focus more on a better rate resolution with higher sensi- 
tivity and less on the measurement range. The sensor bandwidth can be set 
up to 100 Hz. The analog signals of the three gyroscopes are converted with a 
MAXI 226 ADC (Maxim Integrated Products Inc.). Noise is significantly 
reduced by oversampling (up to x 32) within the ADC. The whole inertial 
measurement unit is controlled by an AT90CAN128 microcontroller (Atmel 
Corp.). The on-board CAN controller and the small QFN64 package are ideal 
for miniaturization. With a 32 kB Flash version to come, the prize of the DSP 
will further decrease. 



System Layout 



Miniaturization was the highest priority for designing the new inertial meas- 
urement unit. In order to achieve an extremely small design envelope, we 
chose a six-sided completely closed cubical setup with an edge length of 
1 5 mm. Except for the interface transceivers, all components of the IMU could 
be placed inside the cube (Fig. 2 left). 




Fig. 2. Left: Model of cubical IMU (one side opened for clarity); right: 
Wirelaid PCB 

One main challenge for inertial measurement units is the electrical connection 
of the necessarily more or less orthogonal circuit boards. On most commer- 
cially available IMUs, the boards are connected at their edges with soldering 
pads. However, these connections use several mm 2 on the circuit boards and 
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prohibit further miniaturization. Additionally, they lie open and are not very 
resistant to vibration. For our IMU we developed together with JUMA GmbH 
(Germany) stiff but foldable circuit boards using Wirelaid PCB technology. 
Connections of the rectangular boards are ensured by wires with a diameter 
of 150 mm which are welded onto the top or bottom layer inside the circuit 
board. In contrast to regular PCBs, these connections are maintained when 
folded. But since the board itself does not contain any completely flexible 
parts, the required stiffness for the IMU can be provided. 



3.2 Systems for Gaze Control 

For active gaze stabilization, a high precision camera motion device (Fig. 3) has 
been developed within the FORBIAS project. In a serial gimballed configura- 
tion the camera is actuated around two perpendicular axes. The inner frame 
containing the pan actuator and the pivoted camera is driven by the tilt actu- 
ator containing all Watt Maxon motor (RE-max 24) combined with a backlash 
free HarmonicDrive HDUC-5-50 gearbox with a reduction ratio of 50:1. The 
pan actuator can be less powerful, so a 4.5 Watt motor (RE 16) is used. For the 
control of each axis differential encoders with 512 steps, i.e. 2048 quadcounts 
per revolution, provide sufficient resolution. 




Fig. 3. Camera motion device, shown in a two-degree-of-freedom configu- 
ration 

In this configuration, each axis can be controlled within tolerances of less than 
0.05°. The camera module (Siemens VDO) features a CMOS sensor with an 
effective resolution of 750x400 pixels, 11 mm pixel size and a global shutter. 
The lens shown has a focal length of 16 mm. 
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For further miniaturization of the gaze control, a completely different hard- 
ware is in the process of development (Fig. 4). In the future, we will not actu- 
ate the camera itself but a small mirror in front of it. The actuator has a design 
envelope with edge lengths of less than 65 mm (not including the camera 
itself). 




Fig. 4. New concept for miniaturized gaze control 



3.3 Biomimetic Algorithms 

The control algorithm for inertial camera stabilization is based on a computa- 
tional model [2] of the angular vestibulo-ocular reflex (VOR, Fig. 5). The 
vestibular organ (IMU), in this case the semicircular canals, senses rotations of 
the head (car). In a direct pathway, an inverse stabilization command is sent to 
the eye's muscles (camera motion device). The direct pathway, even though it 
is very rapid due to only three neurons being involved, is not very accurate, 
because it cannot compensate for any changes in the system, e.g. due to devel- 
opment or aging. Therefore, a second parallel pathway through the cerebel- 
lum is continuously adapted to achieve perfect gaze stabilization. Vestibular 
signals, efference copies (feedback) of eye muscle commands and visual sig- 
nals are sent to so-called parallel fibers in the cerebellum. A teaching signal 
containing visual and feedback information is formed in the inferior olive 
(lower brainstem) and passes through climbing fibers to the Purkinje cells. In 
each Purkinje cell, input from several thousand parallel fibers are combined 
for one output, using one climbing fiber teaching signal. 
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Fig. 5. Simplified outline of involved in adaptive vestibular gaze stabiliza- 
tion (VOR) 



The task of the vestibulo-ocular reflex can be understood as active nose con- 
trol [3], with the noise being the unwanted head movement which contami- 
nates the visual signal. The noise is measured independently from the signal 
by the inertial sensors, and is then removed from the signal by actively mov- 
ing the eyes in opposite direction. The adaptive component can, as first 
approximation, be seen as equivalent to what is known in engineering as fil- 
tered-x LMS algorithm [3]. Fig. 6 shows the block diagram of the gaze stabi- 
lization algorithm, assuming equal dynamic behaviour for each actuated axis. 
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Fig. 6. Block diagram of biomimetic control for gaze stabilization. See text 
for explanation. 

In the technical system, motion of the car is sensed by the gyroscopes of the 
inertial measurement unit S. The output of the "brainstem" B corresponds to 
the direct path and is calculated as 
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b = JSco 



(i) 



with the nominal diagonal sensitivity matrix S and the Jacobian J, projecting 
angular rates co on the degrees of freedom of the camera motion device. 
Integrated in I , this part models the direct pathway in the neurological system. 

In the adaptive path, angular rates co are decomposed (D) into "parallel fibre" 
signals pf. In our case, we chose to use measured angular rates and their deriv- 
atives, similar to a notch filter. 



pf = 




( 2 ) 



The basic idea of the adaptive algorithm is decorrelation of gyroscope signals 
and residual optic flow. Decorrelation control has already been proposed as the 
underlying mechanism in adaptation of the biological VOR [4]. Residual optic 
flow consists of three different parts: flow induced by vehicle rotation, trans- 
lation of the camera (driving the vehicle) and changes in the outside world like 
self motion of other cars, pedestrians etc. For understanding the algorithm, it 
is important to know that only the first part, the vehicle's own rotation and the 
corresponding optic flow (OF) is correlated to gyroscope signals. As soon as the 
algorithms manages to decorrelate OF from gyro signals, or better from their 
decompositions pf, residual optic flow is no longer predictable from inertial 
angular rate measurement and therefore comes from either the vehicle's 
translation or changes in the outside world. 

As outlined below, flow caused by vehicle translation can be reduced by means 
of visual tracking (smooth pursuit) of space-fixed targets. Visual tracking also 
allows the camera to follow moving targets, e.g., other vehicle, thereby reduc- 
ing local flow confined to the target. 

In the de correlator, all incoming decomposed signals are adaptively combined 
for each nominal axis. The underlying idea of adaptive linear combination is 
very close to the biological system with Purkinje cells extracting information 
from the parallel fibres. Due to the use of all gyroscope signals for adaptive 
adjustment of each axis, possible non-orthogonalities can be compensated for, 
thereby reducing the required tolerances and thus significantly reducing fab- 
rication costs. 
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The proportional output of the decorrelator is added to the brainstem output 
b and can be expressed as 



p = J[w p w d ] 



'E 3x3 

0 3x3 



0 3x3 

0 3x3 



pf ,w = [w p w d ] ER 3 



(3) 



with co p and co d corresponding to weights of angular rates and their derivatives 
respectively. The output of the derivative path is fed into the system after the 
Integrator I with 



d = J[w p w d ] 
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(4) 



Note that dco/dt is not used for either d or p directly but is needed for weight 
updating, and has thus to enter the decorrelator. For weight update, an anti- 
hebbian learning rule is used 



w = -f pf T a = — 



[co x co y co z dco x /dt dco y/ /dt dco z /dt] 



PpE 3 



0 3x3 P d -E 3x3 



(5) 



where f is the OF projected into the coordinate system of the IMU. /3 p and /3 d 
are learning rates of the proportional and derivative path. Following the idea 
of filtered-x LMS algorithms, pf could be pre-filtered with an estimate of the 
plant's transfer function before the weight update. Laboratory experiments 
showed that this pre-filtering does not significantly improve the performance 
in the frequency range of gaze stabilization. 

As mentioned above, inertial camera stabilization reduces overall optic flow 
caused by unwanted rotational motion and thus removes noise from the 
images. However, camera stabilization alone would not be suitable for collect- 
ing visual information, since perfect stabilization would not keep the line of 
sight on the road. We therefore implemented rapid changes of the camera's 
viewing direction, called saccades in biology [2]. These saccades may occur 
either when the line of sight is too far from the car's long axis (equivalent to 
vestibular nystagmus) or when a novel target should be looked at (equivalent 
to voluntary saccadic gaze shifts). As in the biological system, saccades and 
stabilization of the line of sight interact seamlessly and, most importantly, 
without affecting the efficiency of the adaptive adjustments. 
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Especially when using telephoto lenses, camera stabilization together with sac- 
cades are not sufficient for a driver assistance system, since closer inspection 
of road markers, other vehicles, etc., necessitates yet a third type of camera 
motion: visual tracking. Again, the basic principles can be adopted from biolo- 
gy: primates developed the smooth pursuit system for visual tracking. The 
smooth pursuit system is intimately linked to the VOR system, and exploits the 
same neuronal structure as used for VOR adaptation, the cerebellum, as final 
pathway to convey visual information to the eye muscles. Inspired by nature's 
solution, we implemented visual tracking in our system [5]. The problem of 
visual processing time in the order of 40-100 ms, which could cause instabili- 
ty of the tracking feedback loop, was solved by internally reconstructing tar- 
get motion in space using a delayed combination of camera movement signal 
and IMU data. Fusion of tracking and stabilization commands was based on 
the simple consideration on how target velocity in space f s , camera velocity in 
the car c c , and car velocity in space co (measured by the IMU) interact. Camera 
velocity in space can be expressed as c s = c c + co . Thus, the error between cam- 
era velocity and target velocity e= f s - c s = t s - c c - co can be used to drive cam- 
era motion. Note that, for simplicity, the equations are given for the one- 
dimensional case. 

Visual tracking may, at least in the low frequency range, take over camera sta- 
bilization, thus compromising the success of the adaptive algorithm outlined 
above. Therefore, either adaptation or tracking was switched off in the evalu- 
ations shown below. However, a possible biomimetic solution has already been 
proposed [6] which is based on adaptation at multiple sites. Compared to other 
technical stabilization systems proposed on the basis of biological analogy [7], 
the present systems offers transparent algorithms together with flexibility 
and the same variety of camera movements as known from biology. 



4 Experimental Evaluation under Laboratory Conditions 

For experimental evaluation under reproducible conditions, a test setup with 
the Steward-Gough platform of the Institute of Applied Mechanics at the TU 
Munich has been chosen. The inertial measurement unit and the camera 
motion device are bolted to the upper platform (Fig. 7). The hexapod's tool cen- 
tre point (TCP) is in the middle of the upper platform and was held constant 
during all experiments. The platform's orientation is defined with cardan 
angles a, /3 and y. Due to the 2-dof-setup for camera motion, a was held at zero 
while /3 and y were sine-waves with 12.5° amplitude and a frequency of 3 and 
4.5 rad/s respectively. This motion led to sensed angular rates of approximate- 
ly 100°/s. 
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Fig. 7. Experimental setup on Steward-Gough platform 

With the camera CCD being approximately 60 mm away from the stable TCP, 
translational displacements are limited to about 25 mm. The image from the 
analog camera with telephoto lens (focal length 16 mm) was digitized with a 
frame grabber (Bt787 chipset) and used for online optic flow calculation which 
was the teaching signal in these test. 

We performed two different types of tests. In one series the algorithm started 
without any knowledge of the sensory system and the dynamic performance 
of the actuating unit. The gaze stabilization system learned to identify the 
three rotational sensors, their non-orthogonality and sensitivity. Additionally, 
the inverse of the system's transfer behaviour was approximated. 



Adaptation Par Gain Adaptation Par Derivative Path 




Fig. 8. Adaptation of sensitivity (left) and derivative path gain (right) for 
gyro corresponding to pan axis (in neutral position) 

In a second series we started adaptation with nominal sensor characteristics, 
however non-orthogonality, errors in sensitivity from nominal values and 
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transfer behaviour still had to be compensated for. Fig. 8 (left) shows the 
adjustment of sensitivity. Although the difference to nominal sensitivity was 
only 0.75%, the corresponding difference in angular rate of about l°/s was 
very disturbing and recognizable with a telephoto lens and an aperture angle 
of less than 15° in any direction. As explained above, the derivative path com- 
pensates for the transfer behaviour of the system approximated to have one 
pole. 

At the beginning of adaptation, optic flow of the pan (horizontal) direction 
exceeded 6 pixel/frame (Fig. 9). The subjective impression of the camera video 
was quite bad, the video could hardly be used for any further purposes. After 
one minute of adaptation, residual optic flow reduced significantly to about 2 
pixel/frame. After 2-3 more minutes, optic flow was down to the sub-pixel 
range. Image processing mostly detected 1 pixel/frame every few seconds. For 
the spectators, the video of the camera seemed completely stable. 



Averaged Optic Flow Pan 




time [s] 

Fig. 9. Residual optic flow for pan axis 



5 Verification in Road Trials 

The adaptive system as introduced in chapter 3.3 and evaluated in chapter 4 
was tested in road trials. With adaptation together with visual tracking being 
still in the phase of implementation, the parameters for the camera stabiliza- 
tion system were adapted under laboratory condition with a stable line of sight 
(LOS). 

The inertial measurement unit and the camera motion device were mounted 
in the test vehicle as shown in Fig. 1. An additional camera with wide angle 
lens was used for the general overview of the environment and to detect a 
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point of interest for the camera with telephoto lens. The point of interest was 
coded in platform coordinates and defined the primary reference for the LOS. 
The camera was stabilized around this reference using the inertial measure- 
ment unit as described above. If the difference between the desired LOS and 
the camera axis was more than 1.5°, a saccade was initiated. For visual track- 
ing, target velocity in space was fused with IMU data so that the point of inter- 
est remained stable on the telephoto camera. The required target position and 
velocity which were determined by combining visual information with vehi- 
cle velocity and yaw rate, were sent via the framework's CAN interface to the 
gaze control system. Since translatory vehicle movement is known to the driv- 
er assistance systems, tracking of space fixed targets was considerably 
improved by fusing visual signals collected by the camera with information 
about vehicle motion. 





Fig. 10. Road trials with static wide angle camera (above) and inertially sta- 
bilized camera with telephoto lens 

During the tests, three different modes were evaluated. In mode 1, a lane 
marker about 60 m ahead was chosen and then followed with the stabilized 
camera until it reached a distance of 20 m. Then the next marker was select- 
ed. In a second series, lane separation in a constant distance of about 40 m was 
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focused. In a third mode, no target was defined but the camera was stabilized 
around a constant line of sight. 

In all modes, bumps in the road, as well as low frequency pitch could be well 
compensated for. Especially when driving in the dark, the inertially stabilized 
camera significantly improves the readability of road signs. While with longer 
exposure times indications are blurred in images of the wide angle camera, the 
camera with telephoto lens showed well readable writing on the signs (Fig. 
10). This is especially remarkable since the effects of motion are more adverse 
when using lenses with longer focal lengths. 



7 Conclusions 

This paper introduced an inertially stabilized camera platform for driver assis- 
tance systems with the focus on biologically inspired multi-sensor fusion of 
data from an inertial measurement unit and image processing. Adaptive algo- 
rithms have been developed to compensate for non-orthogonality of sensitive 
sensor axes and deviations from nominal sensor characteristics due to loose 
manufacturing tolerances and variability of low-cost MEMS sensors. This helps 
reducing the cost of actively stabilized camera systems significantly since the 
use of expensive sensors and highly precise fabrication is not necessary. The 
system showed its capacities and improvements to current technology in first 
test drives. With a new miniaturized version of a gaze manipulation device 
being available soon, this can be the basis of driver assistance systems. 
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Low Speed Collision Avoidance System 



J. Kibbel, H. Salow, M. Dittmer, IBEO Automobile Sensor GmbH 



Abstract 

Most collisions occur with ahead driving vehicles or at speed below 
30 km/h, respectively. Driving in columns is tiring and can lead to 
inattention and distraction and is therefore one of the main sources 
of accidents. A Low Speed Collision Avoidance System can help the 
driver avoiding these kinds of accidents. 



1 Introduction 

Driving in urban areas can be a challenge not only for inexperienced drivers. 
The driver is distracted by information from inside the car like for example 
messages from the entertainment system, the navigation system, the tele- 
phone, the radio or passengers. Additionally the driver has to deal with all the 
information from outside the car like traffic signs or the behaviour of other 
road users. Just a moment of distraction is enough to cause a collision on city 
streets. The Low Speed Collision Avoidance System can help the driver avoid- 
ing these accidents by analyzing the area in front of the car. If an upcoming 
crash is detected the system will brake the car automatically to avoid the col- 
lision. 



2 The Measurement System 

The ALASCA XT Laserscanner used for environment detection is the latest 
member of IBEO Automobile Sensor GmbH Automotive Laserscanner family. 

This Laserscanner can be integrated in the vehicle behind a plastic by consid- 
eration of the design of the vehicle. The sensor offers a horizontal field of view 
of up to 240° with an angular resolution between 0.1° and 1.00°, depending 
on the scan frequency, which can be modified from 12.5 Hz to 25 Hz during 
operation. The radial range for object detection is from 0.30 m up to 200 m. 
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The four scan planes obtain a vertical opening angle of 3.2° in the vertical 
direction. This allows a compensation of the vehicle pitch angle, as shown in 
Fig. 2. 




Fig. 1 . ALASCA XT multilayer Laserscanner 

For more information on Laserscanner and their operating principle see [1]. 

Based on the measured scan data a tracking algorithm detects objects and cal- 
culates their velocity, size and class [2]. With these detailed information about 
the surrounding of the car many application can do individual things like col- 
lision mitigation. 




3 Low Speed Collision Avoidance Application 

The Low Speed Collision Avoidance System assists the driver to avoid rear end 
collisions in urban areas. To detect an upcoming accident a monitored area is 
calculated. The length of this area is depending on the ego velocity and on the 
velocity of a potential collision object. It consists of three parts (Fig. 3), the 
safety distance, the system delay and the braking distance. 
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The first part is the minimum distance between the ego vehicle and an object 
after the braking, the safety distance. To optimize the distance several test 
drives were made with different drivers. These tests showed that if the safety 
distance is to small the driver have less confidence in the system. If the dis- 
tance is too large the system may brake the car even if the driver feels that he 
could still master the traffic situation. It seems that a distance of about one 
meter is the distance that is accepted by most drivers. 



The delay time of the braking system is responsible for the second part, the 
system delay s d . Even if the delay time of modern brakes was reduced in the 
last years, it still takes some time for the brakes to achieve the full decelera- 
tion that was requested by the system. The system delay s d depends on the 
velocity of the ego car v ego and on its maximum deceleration a dec : 



s d = 



V ego + ■ 



a 



dec 



\t 

SystemOjfset SystemOjfset 



(1) 



The third part, the braking distance offset s h , is depending on the ego velocity 
and the velocity of a potential collision object. If we assume that the car in 
front has the same maximum deceleration than the ego vehicle but a lower 
velocity v 0h •, we have to compensate the shorter braking distance of the car in 
front. The braking distance offset is the difference between the braking dis- 
tance of the ego vehicle and the braking distance of a potential collision object. 
Because the velocity of the ego vehicle is already reduced during the system 
delay time, the offset can calculated as follow: 
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Furthermore, the direction of the monitored area is depending on the front 
wheel angle. If an object like a car, truck, pedestrian or bike is entering the 
monitored area, the car is decelerated as long as the object is located within 
the monitored area to avoid a rear-end collision accident. If the collision object 
is leaving the monitored area or if the driver sidesteps the object, the braking 
is discontinued. Test drives showed that triggering the brakes puts the atten- 
tion of the driver to the critical area in front of the car so that the driver can 
remedy the critical situation without any further support. 
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Fig. 3. Field of view of the Laserscanner (grey). Monitored area consisting 
of: Minimum distance to target after braking, the safety distance 
(green). System delay, depending on the vehicle velocity (orange). 
Braking distance offset depending on the ego velocity and the 
velocity of a potential collision object (red). 



4 Typical Traffic Scenes 

In the following three typical traffic scenes and the corresponding reaction of 
the Low Speed Collision Avoidance Application are analysed. 




Fig. 4. Critical situation at roundabouts (right). Monitored area in front of 
the car (bottom left). The length of this area is dominated by the 
safety distance (green). The system delay distance (orange) is 
shorter than the safety distance. The braking distance offset (red) 
is much smaller than the safety distance. The time depending 
deceleration of the ego vehicle (top left). 

One of the main sources of rear end collisions is entering a roundabout (Fig. 4). 
A car in front of the ego vehicle seams to enter the roundabout. The driver 
then puts the focus on trying to find a gap in the traffic inside the roundabout 
and starts entering the roundabout. If the car in front is still waiting to enter 
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the roundabout a rear end collision may happen. The Low Speed Collision 
Avoidance Application is monitoring the area in front of the ego vehicle and 
because of the low velocity of the car this area is dominated by the length of 
the safety distance. The driver can still get very close to vehicle in front, to 
observe the roundabout, without being disturbed by the system. Not until the 
driver is getting to close, the system will brake the vehicle to avoid an acci- 
dent. 

If the driver is distracted in urban areas he could easily oversight for example 
a traffic jam (Fig. 5). In this situation the Low Speed Collision Avoidance 
System could put his attention to the critical situation in front of the car. Then 
the driver has the possibility to control the situation on his own by braking the 
car or by giving way. Otherwise if the driver does not react appropriate the 
system will brake the car as long as there are objects inside the endangered 
area. Because of the relative low difference between the velocity of the ego 
vehicle an the potential collision vehicle the braking distance offset is small 
and the system delay distance is dominating the length of the monitored area 
in front of the car. 
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Fig. 5. Critical situation with a slow driving vehicle in front of the ego 
vehicle (right). Monitored area in front of the car (bottom left). The 
length of this area is dominated by the system delay distance 
(orange). The safety distance (green) and the braking distance off- 
set (red) are shorter than the system delay distance. The time 
depending deceleration of the ego vehicle (top left). 

A very dangerous situation can occur if the driver is approaching a stationary 
vehicle without noticing it (Fig. 6). Because of the huge different velocity of 
both cars the monitored area is much longer than in the previous example and 
is equivalent to the situation where no object is detected in front of the car. If 
a stationary object is detected in front of the car the Low Speed Collision 
Avoidance System hast to trigger the brake much more in advance. If the sys- 
tem detects that the driver is aware of the critical situation the control of the 
brakes is put back to the driver. To detect the alertness of the driver, the move- 
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ment of the steering wheel as well as changes at the accelerator or the brake 
pedal are analysed. 




Fig. 6. Critical situation with stationary vehicle in front of the ego vehicle 
(right). Monitored area in front of the car (bottom left). The length 
of this area is dominated by the braking distance offset (red). The 
safety distance (green) and the system delay distance (orange) are 
shorter than the braking distance offset. The time depending 
deceleration of the ego vehicle (top left). 



5 Conclusion and Outlook 

The Low Speed Collision Avoidance System has the potential to decrease the 
number and the costs of rear end collision in urban area dramatically. To avoid 
a collision the system triggers the brakes if a dangerous situation is detected. 
By triggering the brakes the attention of the driver is put to critical area in 
front of the car. In most times the driver then takes control and can avoid a col- 
lision either by giving way if possible, or by braking the car. If the driver is not 
reacting appropriate the system will bring the car to a halt or releases the 
brakes if the object in front left the endangered area. ' 
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Abstract 

Driver assistance systems for both passenger cars and commercial 
vehicles have made much progress throughout the past few years. 
Laserscanners have proven to meet the requirements for serving as 
their sensor backbone. This article focusses on the potential of auto- 
motive laserscanners supporting advanced driver assistance systems 
on passenger cars and trucks. 



1 Introduction 

The introduction of active and passive automotive safety systems has signifi- 
cantly contributed to the reduction of traffic accidents and injuries. 
Nevertheless european road traffic takes a toll of 40000 lifes every year. The 
next step to increase traffic safety will be taken by the introduction of 
advanced driver assistance systems as proposed by the European Comission. 
Its e-safety initiative [1] sets the goal of reducing the number of traffic acci- 
dents by the year 2010 by 50%. Applications like distance alert, lane guiding 
assist, collision mitigation, or drowsy driver warning assist will help to reduce 
the consequences of crashes for all collision partners. Before this background 
advanced driver assistance systems for passenger vehicles have been receiv- 
ing much attention throughout the past few years. Less noticable, advanced 
driver assistance systems for trucks have also made much progress. For the 
general acceptance of advanced driver assistance systems reliable sensors 
with a high degree of availability -even under adverse environmental condi- 
tions- are indispensable. Infrared laserscanner technology stands in the first 
line amongst the sensor technologies of the choice. 

This article focusses on the applications of laserscanners, developed by IBEO 
Automobile Sensor GmbH. Their capability of a high range finding distance 
of up to 200 m in combination with a fine horizontal resolution make it a 
favourable technology for applications in classic ACC systems in combination 
with applications like Stop & Go including automatic restart. Lane Departure 
Warning, Pedestrian Protection, Pre-Crash, and Automatic Emergency Brake. 
For commercial vehicles the applications Slow Moving Assist, Turning Assist, 
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and Cut-In Assist are of predominant relevance. All mentioned applications 
can be operated by one single laserscanner system. 



2 ADAS Applications Overview 

The ALASCA XT laserscanner series developed by Ibeo Automobile Sensor 
supports up to seven ADAS applications each, on passenger cars and commer- 
cial vehicles. While some of them like Automatic Emergency Brake and Lane 
Departure Warning are common for both vehicle types the Turning Assist and 
the Slow Moving Assist are specifically designed to serve the needs of com- 
mercial vehicles. An overview of the applications is presented in Fig. 1. 




Fig. 1 . ADAS applications for passenger cars (top) and trucks (below) sup- 
ported by a single Laserscanner ALASCA XT system 

In the following sections laserscanner performance under adverse weather sit- 
uations will be addressed and specific attention will be paid to a selected set 
of ADAS applications on passenger cars and commercial vehicles. 
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3 Environmental Robustness of the Laserscanner System 

The ALASCA laserscanner series developed by Ibeo Automobile Sensor was 
designed such that adverse weather conditions would not inhibit a reliable 
performance of the application. Specifically tests with dense traffic on express- 
ways under heavy rain fall conditions prooved full functionalty (see Fig. 2). 
Problems of blooming, as known from video technology are unknown to the 
laserscanner because it carries its own light source and sunlight is removed by 
an adequate filtering technique. Also there exists no danger of being blinded 
by other vehicles which themselves are carrying infrared sensors. The receiv- 
er characteristics of the laserscanner matches at the worst only for the frac- 
tion of a second the transmitter characteristics of the sensor on the other car 
which results in the drop-out of a small number of measurement points. These 
points can easily be removed by a software filtering process and do not affect 
the performance of applications. 




Fig. 2. Expressway driving szene under heavy rainfall conditions. Speed = 

82 km/h. The main frame shows the laserscanner data and the 
tracked vehicles (objects). The top left insert shows the video mon- 
itor. 

The performance of the ALASCA system was tested in Sweden during winter 
conditions. All automotive requirements were met. Fig. 3 shows a screen shot 
of a test driving situation where a passenger car is detected by the ALASCA 
XT system at a distance of 225 m. 
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Fig. 3. Screen shot of a driving situation during winter test in Sweden 
where a passenger car at 225 m ahead is picked up by the laser- 
scanner system. 



4 Vehicle Integration 

The integration of a laserscanner into a vehicle includes two aspects of work. 
The mechanical part must ensure that the sensor can be accommodated with- 
in the contour of the vehicle body. The second part consists of its integration 
into the electronic network. Via the vehicle CPU the ALASCA applications 
transmit their commands, e.g. to the throttle or to the brakes of the vehicle. In 
the reverse direction the ALASCA CPU continually receives vehicle data, e.g. 
steering angle, yaw-rate, speed, and brake pressure. 

For developmetal purposes a recording system, consisting of a video camera 
and a computer, may be connected. The entire data stream of the ALASCA sys- 
tem together with the video image may be stored for later evaluation purpos- 
es. Meanwhile a number of vehicles from different manufacturers has been 
equipped with an ALASCA system by IBEO AS. Depending on the develop- 
mental requirements either a single central sensor with a horizontal opening 
angle of 150° or a sensor pair with 240° was built in. In the latter case the 
ALASCA data are fused in order to provide a unique stream of commands to 
the vehicle system. 
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5 Applications 

5.1 ACC with Stop & Go 

The first Adaptive Cruise Control systems (ACC) were introduced about 10 
years ago as comfort systems in upper class passenger cars. However they 
only reached a relatively small market penetration. Starting at a minimum 
speed of about 30 km/h these radar-based systems control the speed depend- 
ing upon distance and speed of the preceding vehicle. Below the minimum 
speed a control function is not available due to the fact that the low horizontal 
resolution of radar-based sensors does not permit a safe surveillance of the 
immediate proximity of the car. 

With the new ALASCA XT it has become possible to simultaneously run long 
range and short range ADAS applications. Adaptive cruise control can now be 
operated within the full speed range from low speed city traffic (including full 
stop) to high speed expressway traffic. While for ACC a range of up to 200 m 
is of predominant importance. Stop & Go requires the wide horizontal scan 
angle paired with a fine angular resolution. This feature ensures an early 
detection of cut-in and cut-out procedures which results in a significant 
increase in comfort and safety for controlling the longitudinal speed. After an 
automatic stop the system has the capability of evoking a safe automatic 
restart because there exists sufficiently dense information about objects in the 
immediate proximity of the car. 




Fig. 4. Schematic of laserscanner supported full range ACC with Stop & 
Go. The integrated laserscanner ALASCA XT detects objects up to 
a distance of 200 m in a horizontal scan area of 150°. 



A driving szenario with a laserscanner in a typical Stop & Go situation is 
shown in Fig. 5 as seen by the measurement system. The left part of Fig. 5 
shows the screen shot of the laserscanner data from a bird-view perspective. 
The right part of Fig. 5 exhibits the corresponding video image which serves 
for reference purposes only. 
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Fig. 5. Driving situation with ALASCA and Stop & Go operation. The left 
side shows the screenshot of measurement data in bird-view per- 
spective. The right side shows the corresponding video image 
which has been taken for reasons of reference only. Note: The video 
image only partly shows the center part of the laserscanner screen- 
shot data. 




Fig. 6. Compilation of laserscanner data in a real traffic jam Stop & Go 
szenario on an expressway. The red trace (d) shows the distance to 
the leading object, the blue trace (v) the ego-velocity, and the pink 
trace (a) the ego-acceleration. 

In order to demonstrate performance in a real Stop & Go situation the test 
vehicle was driven during a traffic jam on an expressway. Fig. 6 shows a com- 
pliation of the ALASCA XT data for a sampling time of 110 sec. The red trace 
shows the distance to the leading object, the blue trace is the test vehicle ego- 
velocity and the pink trace denotes the ego-acceleration. It can be seen that 
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the test vehicle controls its velocity according to the distance of the leading 
object. The parameters are set such that a minimum distance of about 5 m is 
kept when traffic comes to a stand still. Acceleration for the restart is evoked 
with a slight delay after the distance to the leading object has already 
increased again. Deceleration down to zero velocity is executed when the dis- 
tance approaches the set limit of 5 m. 



5.2 Pedestrian Protection 

The e-Safety initiative of the European Commision proposes to cut the number 
of road accidents by 50% by the year 2010 [1]. Specifically Pedestrian 
Protection systems will become an essential safety function in future vehicles 
as legislation forces automotive manufacturers to improve pedestrian safety. 
For such systems the ability to detect, track, and classify pedestrians is manda- 
tory. 

The accident analysis describes the car-to-pedestrian accident mainly as a 
front crash with a moving pedestrian. The front crash of a car with a pedes- 
trian is associated with about 70% of all car-to-pedestrian accidents. In addi- 
tion 94% of the pedestrians involved are moving [2]. These accidents can be 
predicted by the ALASCA system which corresponds to 66% of all car-to- 
pedestrian accidents. As a safety measure in order to protect pedestrians in 
the moment in which an accident becomes inevitable, i.e. an active bonnet can 
be triggered such as presently realized by means of a bumper integrated 
impact sensor [3]. 

The strategy to classify pedestrians with a laserscanner is based on the fol- 
lowing concept. An object with the shape of a pedestrian is classified as a 
pedestrian only if the object is moving or has moved during the period of 
tracking in the past [4]. This is a good assumption, as 94% of the pedestrians 
involved in car-to-pedestrian accidents are walking or running [3]. Otherwise 
the object will be detected as a non-moving obstacle. As long as a small obsta- 
cle is not moving towards the road, there is no risk of collision, neither for the 
vehicle nor for the object itself. In order to increase the robustness of the 
pedestrian classification, the typical oscillatory movement of the legs in the 
range image sequences and the history of the tracked object with respect to 
previous classification results are additionally taken into account [5]. 

If the pedestrian is detected in a well defined region in front of the car, the so- 
called region-of-no-escape (RONE), the car-to-pedestrian accident is defined 
inevitable (see Fig. 7). Every pedestrian colliding with the car's frontal region 
will enter the RONE in a certain time before the accident. This can be moni- 
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tored using Laserscanner data. The dimensions of the RONE are determined 
under consideration of the maximum deceleration and acceleration capabili- 
ties of the ego vehicle and the pedestrian, respectively. The potential initial 
acceleration of a pedestrian has been evaluated in detail. It was found that typ- 
ical dimensions of the RONE are 1 m to 3 m in front of the vehicle, depend- 
ing on the velocity of the vehicle and the pedestrian. The determination of the 
RONE parameters - in addition to pedestrian classification algorithms - pro- 
vides a strategy to predict an impending car-to-pedestrian accident with a 
very high reliability. 




Fig. 7. Schematic of the region-of-no-escape (RONE) in front of a car 

The verfication results are generated with recorded real data of a laserscanner 
integrated in a passenger car, with a total driven distance of more than 10,000 
km on different road types. The first results show a false alarm rate of 0.7 per 
100 km for the Pedestrian Protection application, without use of the analysis 
of the oszillatory leg motion pattern. Adding the leg motion analysis first 
results indicate that a false alarm rate of 1 per 10 8 km should be possible. 

The described system therefore shows the potential of an early detection of 
66% of all car-to-pedestrian accident scenarios corresponding to 94% of pedes- 
trian front accidents. In conjunction with active safety elements it offers the 
possibility to mitigate the consequences of car-to-pedestrian accidents. 



5.3 Lane Departure Warning 

Long rides under monotonous conditions may lead to a sudden loss of alert- 
ness, the so-called microsleep. This situation bears the risk of unintentional 
lane departure with the result of causing an accident. A Lane Departure 
Warning (LDW) system greatly reduces this risk. The task of LDW is to alert 
the driver before an unintended lane departure takes place. Video based sys- 
tems [6] which monitor the lane markings ahead of the vehicle and infrared 
systems [7] measuring the markings straight down, are already commercially 
available. 
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An ALASCA system already built into the vehicle for supporting one or more 
of the applications mentioned above is capable of additionally supporting LDW. 
Additionally to monitoring the objects on the road in the proximity of the car 
ALASCA also measures the position of the lane markings. When the vehicle 
starts drifting off the lane an adequate warning signal is issued to the driver 
enabling him to prevent an accident by steering back into the lane. 

Technically the LDW system is realized by utilizing an unused fraction of the 
240° ALASCA scan angle. A mirror is positioned within the integration cham- 
ber such that an additional detection area on the road surface is created. This 
area is located at a distance of about 4 m ahead of the vehicle (Fig. 8). 
Different types of lane markings of varying quality can be recognized. 
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Lane Departure Warning right! 






Fig. 8. Lane Departure Warning with a laserscanner system. The schemat- 
ic (top) shows the beam directions for both, the standard set of 
beams and the additional LDW-beams. A corresponding driving sit- 
uation is shown below. 



5.4 Slow Moving Assist and Turning Assist for Trucks 

Unlike cars, trucks have much larger front and side blind spots than passenger 
vehicles. Vulnerable road users located in these blind spots are exposed to a 
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high risk of accident. This situation is the reason for hundreds of fatal acci- 
dents per year only on German roads. These accidents occur in urban traffic 
situations such as pictured in Fig. 9. European legislation requires from 
January 2005 on an additional mirror as a first countermeasure. Eventhough 
mirrors are an easy-to-install low-cost solution they are often inadequate to 
provide a proper view when turning or when starting to move. The reason is 
that first some blind spot zones remain and second the number of mirrors (up 
to 6), the truck driver must supervise, may become too large for a fast and intu- 
itive information processing. This is where the ALASCA laserscanner provides 
a suitable ADAS alternative. With its large scan angle and its fine horizontal 
angular resolution it is particularly suited for monitoring the entire blind spot 
area. A situation like the one shown in Fig. 9 either leads to a driver warning 
or, in case of an imminent collision, braking and disabling of the accelerator. 




Fig. 9. Top view on a typical situation where high alert of the slow mov- 
ing assist and turning assist is required. A truck is waiting at a traf- 
fic light with a bycicle located in its blind spot. The arrows indicate 
the intersecting paths of the two vehicles after the green light 
comes on. The ALASCA system prevents the accident at the red 
star-marked area. 

The automotive laserscanner ALASCA was integrated into the right side of the 
front bumper of a commercial truck. The integration chamber harmonically 
fits in with the contour of the vehicle. The available scan angle is about 220° 
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such that monitoring of the immediate proximity of the areas in front of the 
truck and on the right side of the truck are warranted. Vulnerable road users 
like byciclists or pedestrians entering the blind zones can now safely be 
detected. Warning signals can be issued and a collision will be inhibited. 

A screen shot of a driving szenario is shown in Fig. 10. It can be seen that the 
wide scan area has been realized and that pedestrians are detected near the 
truck as well as at larger distances. 
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Fig. 10. Screen shot of a driving situation with commercial truck and inte- 
grated Laserscanner in a narrow road environment. Note the 
extreme wide scan area. The ALASCA system detects and classi- 
fies the pedestrians (denoted Ped. 1 to 4) within its monitoring zone 
and thus provides the required safety function. 

Once the truck is equipped with the ALASCA there exists the option of also 
serving the other ADAS applications as mentioned above (Fig. 1). Their imple- 
mentation is basically a matter of the dedicated software and the application 
specific HMI. 



5.5 Cut-In Assist for Trucks 

Another important safety application is the Cut-In Assist for trucks. This assist 
is designed to support drivers operating their vehicles on expressways. After 
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overtaking slower traffic the subsequent cut-in procedure frequently leads to 
dangerous situations and to accidents due to a misjudgement by the driver of 
the available gap on the slower lane. Here the laserscanner system supports 
the driver when moving back into the main lane (see Fig. 11). It outputs a Go- 
Signal when length and position of the gap in the right lane permits a safe lane 
change. Otherwise the driver is prompted with a No-Go-signal. 




Fig. 1 1 . Schematic of the application Cut-In Assist for commercial vehicles. 



6. Conclusion 

The ALASCA sensor integrated into the demonstrators showed the ability of 
operating a wide range of ADAS applications on passenger cars as well as on 
trucks. The combination of a long detection range of up to 200 m and a large 
horizontal scan angle of up to 220° provides the opportunity to simultaneous- 
ly operate seven applications with one single sensor system. 
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Abstract 

Inter-vehicle communication has been one of the most emerging 
fields of interest in the automobile industry during the past few 
years. It is considered as one key enabler for active safety and fore- 
sighted driving assistance services. A very prominent example is a 
local danger warning service. Vehicles instantaneously inform others 
about critical situations like aquaplaning, accidents or obstacles on 
the road. The detection of such environmental conditions should be 
accomplished by means of already available and widely deployed on- 
board sensors. In this paper we present a new efficient and light- 
weight approach to estimate the existence of an obstacle, based only 
on the vehicles' driving dynamics. The approach is evaluated in 
terms of contribution to the application domain, operability and reli- 
ability. A prototype implementation is described and first results 
from real track driving tests are presented. 



1 Introduction 

Today, nearly all vehicles are already equipped with sophisticated on-board 
sensors to detect critical driving conditions and take appropriate actions. 
These systems are mostly reactive and only cover the vehicle's current posi- 
tion. They are not able to foresee critical situations, which the vehicle may 
potentially encounter in the future. During the last decade, car manufactures 
and suppliers put a lot of effort into developing foreseeing and proactive driv- 
er assistance systems like ACC. They are mainly based on near range sensing 
technologies such as radar, lidar, ultra-sonic or optical image processing. Some 
of these systems are already available in series products and perform very 
well. Unfortunately, these systems are usually very expensive and only avail- 
able for upper class vehicles. 

The paradigm central to this approach is different. Inter-vehicle communica- 
tion based on vehicular ad-hoc networks will play an increasingly important 
role for future vehicle active safety systems. The main idea is to interlink the 
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on-board sensor systems of vehicles. Thus, vehicles have access to (abstracted 
and standardised) current sensor information from locations different from 
their own positions. This enables the enhancement of the drivers' horizon 
through safety related information that is gathered and provided directly by 
other vehicles. Therefore, inter-vehicle messaging is widely considered as a 
promising complementary technology to support foresighted driving and acci- 
dent avoidance. One of the most promising active safety applications in vehic- 
ular ad-hoc networks is a local danger warning (LDW) service as described in 
[4]. Vehicles directly exchange information about the present road condition 
and dangerous situations. They are therefore able to warn their drivers of 
potential upcoming dangers in a timely manner. This will significantly reduce 
the likelihood that vehicles will get into a critical driving condition, because 
the drivers' attention will be systematically directed to the spot of the danger 
in the run-up of the situation. Obviously, this in turn will lead to a significant 
reduction of accidents. 

This paper covers two key concepts towards a foresighted obstacle warning in 
the domain of local danger warning based on inter-vehicle communication. We 
present a lightweight approach to detect the occurrence of an evasion manoeu- 
vre, which in turn can be used as an indication for a potential obstacle on the 
road that triggers additional subsequent evaluation algorithms. 

The rest of the paper is organized as follows: First we point out the scope and 
objectives of the proposed solution. In chapter three we briefly describe the 
related work and our contribution. Section four describes the proposed 
approach in detail. Later, the prototype implementation and the system archi- 
tecture are outlined. Finally, we will present an evaluation and conclusion of 
our work. 



2 Scope and Objectives 

The scope of this paper is limited to the domain of cooperative local danger 
warning based on inter-vehicle communication. The proposed approach will 
complement other existing and future foresighted driving assistance systems 
that help to avoid critical driving conditions. In contrast to these stand-alone 
in-vehicle systems, cooperative systems rely on information provided by other 
vehicles. Thus, the reliability of the overall system depends on both the relia- 
bility of the communication and the penetration rate of the system. Putting it 
differently, cooperative systems can only operate well if sufficient information 
from contributing vehicles is available. As a consequence, it is unlikely that 
those cooperative systems will (in the near future) take appropriate actions 
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autonomously. Instead, they will systematically direct the drivers' attention to 
dedicated information of particular interest in a certain situation. 
Nevertheless, once deployed in vehicles, they are cheap and easy to integrate 
and complementary to stand-alone sensor systems. In addition, the range is 
basically not limited to a few hundred meters in front of the vehicle, which is 
typically the case for on-board sensor systems. On top, the overall contribution 
and benefit to foresighted active safety will steadily increase over time [2]. 

In order to achieve a quick deployment and high market penetration, the sys- 
tem should be cheap and easy to integrate into the vehicles. In particular, no 
additional new sensor systems should be necessary. Instead, common and 
already widely deployed sensor systems should be used. Vehicles should not 
necessarily need to be equipped with any kind of navigation system or digital 
map. To enable quick market penetration, the system should also be available 
as an additional aftermarket solution. 

For the following considerations, we assume that an ad-hoc communication 
system as well as a cooperative local danger warning service is available in 
some vehicles, and that these vehicles are equipped with a standardised posi- 
tioning system with sufficient accuracy, like GPS or Galileo. 



3 Related Work and Contribution 

In recent years, a couple of national and international research programs and 
standardization committees have been dealing with inter-vehicle communica- 
tion and the dissemination of warning messages [4 - 10]. Basically, they 
focused on communication issues and challenges, but did not address in detail 
the recognition of road obstacles. 

We decided to deduce evidence for obstacles by means of vehicles' driving 
dynamics during an evasion manoeuvre similar to [FOR05]. This approach has 
two big advantages. First, it only depends on common on-board sensors, with- 
out the need of complex additional hardware like radar or cameras. Second, 
assuming a standardized interface to access basic sensor information, the sys- 
tem can be offered as an aftermarket add-on, almost independent of the brand 
and the model of a vehicle. 

To understand the concept, we foremost have to get granular on the technique 
of obstacle detection by vehicle's driving dynamics. First of all we can mark 
out three distinct states a vehicle can reside in during an evasion manoeuvre, 
namely whether the vehicle is moving out of its lane, is driving past the obsta- 
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cle, or is moving back into its original lane. Each of those three states is defined 
by characteristic values of signals, like lateral acceleration, longitudinal accel- 
eration, speed and yaw. 




Fig. 1. Evasion manoeuvre [11] 

As shown in [FOR05], there are some requirements which have to be fulfilled 
during signal analysis to reach a conclusion on the existence of an evasion 
manoeuvre. The most important aspect is the detection of a specific threshold 
of the lateral acceleration, both as negative and positive amplitude in a defined 
period of time, which means that the signal changes from one relative 
extremum to another. This aspect of curve progression is typical for all evasion 
manoeuvres (moving out of lane). However, concerning the varying occurrence 
of each following situation (which is significantly influenced by drivers' behav- 
iour and the present road characteristics) it is hard to make a reliable state- 
ment about these situations by only analyzing this specific sensor information. 
Therefore, we have developed an enhanced but lightweight concept which is 
supposed to increase the reliability of the obstacle detection system. 



4 Obstacle Detection 

In this section we briefly describe the basic idea and workflow of our system 
in addition to some of the decision criteria to presume whether or not an obsta- 
cle has been detected. 
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The proposed obstacle detection consists of two distinct successive phases: An 
initial indication for an obstacle which is deduced from evidence for an evasion 
manoeuvre, and a subsequent verification process. For the first, we will adapt 
the suggestion presented in [11]. For the latter, we propose a systematic eval- 
uation of the trajectory of the following vehicles. Fig, 2 shows the basic control 
flow. 



Phase 1 : Detection of an evasion manouvre 




Fig. 2. Basic control flow 

4.1 Phase 1 : Detection of an Evasion Manoeuvre 

The basic idea for the detection of an evasion manoeuvre is to simplify the 
evaluation of the vehicle's driving dynamics to the first state when it quickly 
moves out of its lane and countersteers again. The resulting characteristic 
curve is typical for all evasion manoeuvres and is basically independent from 
the present steering angle and the dimensions of the obstacle. However, a sim- 
ilar behaviour can be found in steep double curves. In order to distinguish eva- 
sion manoeuvres from similar accelerations caused by these particular road 
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geometries, it has to be verified whether or not the vehicle returns back on its 
designated route (Fig. 4). This consideration is based on the fact that usually 
road geometries do not consist of two successive double bends. 

Thus, the proposed detection of an evasion manoeuvre consists of the follow- 
ing three parts: 

a) Forecast of the most probable route that will be driven by a vehicle in 
the imminent future. 

b) The detection of a sequence of unusual lateral accelerations, typical for 
evasion manoeuvres. 

c) An evaluation of the deviation of the actual trajectory from the fore- 
casted route. 

Since the system is required to be permanently active, there is one state which 
is characterized by regular drivers' behaviour, which means that no critical 
situation is on the road. In this section the car continuously calculates its hypo- 
thetical route whenever it changes its heading (a) according to [Derived from 
the steering angle. If the vehicle is equipped with a digital map or navigation 
system, such estimation is obviously not necessary. Instead, the known road 
geometry can be used.]: 

d 

r = (1) 

tan(y) 



where c/ w is the particular wheelbase of the vehicle and y refers to the present 
steering angle. Having the information of the estimated route, the car can 
build up its own local logical map with its own coordinate system where it is 
located in the origin (Fig. 3). 




Fig. 3. Route estimation 

Besides this route estimation the system concurrently analyzes the received 
sensor information with regard to a specified threshold of the lateral accelera- 
tion signal as described in the preceding section indicating a move out of lane 
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manoeuvre. In case of sufficient evidence for such manoeuvre (b), the route, 
which the car was supposed to reside on before the manoeuvre, will be con- 
sidered as the reference route and a second evaluation will be initiated (c) 
where the real driven route will be compared to the reference one (Fig. 4). 



Fig. 4. Real driven route calculation 

To perform this kind of evaluation, we have to take a closer look at the tech- 
nique how the real driven route can be calculated without any global coordi- 
nate or positioning system. This is necessary because the current systems do 
not meet the requirements of availability and reliability for this kind of appli- 
cation. 

The main idea is the abstraction of a car ride to single steps, where each is 
described by an appropriate orbit. This approach is similar to the calculation 
of the hypothetical route. However, it is more complex, because the car is mov- 
ing in the deduced local map. Thus, it is not positioned in the origin anymore 
and therefore some different calculations have to be done. We will briefly 
describe these calculations in the following paragraphs. 

After initiating the second calculation, the vehicle moves on the pursuant 
orbit, so that the current position has to be calculated. This position P (Fig. 6) 
can be described by the driven distance w, which can be assessed by the speed 
v measured during an appropriate time period. 



On the basis of this data, the angle a and the current position P, where the 
vehicle is positioned on the circle, can be calculated as follows: 



Y 



Real driven route 




V 

W = — 



(2) 






|cos(a-Jt/ 180 )-r\ — - 



P= v ' + M 

^sin(a -jr/ 180 )T y 



(3) 
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where M refers to the centre of the corresponding circle. Considering, that our 
steering angle is changing continuously, the actual route can be calculated by 
getting the radius of the according circle. Therefore, the new centre of the new 
circle has to be calculated. This can be done by defining a direction vector 
from our current position to the old centre and calculate an appropriate unit 
vector. Multiplying the unit vector by the new radius and adding the result to 
the position of the vehicle finally leads to the new centre. 




Y 

esli mated route 

/ 

/ 




Fig. 5. Route estimation Fig. 6. Real route calculation 

(mathematical view) (mathematical view) 

After each position update, the system verifies, whether the vehicle returns 
back on the reference route or not. This verification is done by treating the 
resulting deviation between the predicted and the actual route. If this devia- 
tion still exceeds a certain threshold after a specific time t (which is derived 
from the speed that the vehicle had right before the evasion manoeuvre), the 
existence of an evasion manoeuvre is assumed to be true. To further improve 
the quality of service, both the deviation and the local extremum of the initial 
lateral accelerations can be used to deduce the likelihood for a particular eva- 
sion manoeuvre. 

After running through that phase, the system resets and returns back to the 
first one. 



4.2 Phase 2: Verification Process 

However, an evasion manoeuvre obviously still does not give sufficient evi- 
dence for an obstacle that endangers other road users. Similar manoeuvres 
also occur when, for example, a driver rapidly changes his lane to overtake 
another vehicle. In addition, obstacles often move and disappear again very 
quickly (For example animals, a car door that has been opened carelessly etc.). 
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One very obvious consequence of the existence of a static obstacle that endan- 
gers others is the fact that vehicles will not pass over the respective location. 
This can be exploited for the subsequent verification process. The main idea is 
to systematically analyze the exact trajectory driven by the following vehicles. 
Note that because of performance issues, it is not (yet) possible to reduce the 
obstacle detection mechanism to only this phenomenon. This would require a 
permanent detailed evaluation of all trajectories of all vehicles at any time. In 
addition, all trajectories have to be permanently exchanged among the vehi- 
cles which would quickly lead to a congestion of the available communication 
channels. Again we assume that vehicles are equipped with a positioning sys- 
tem. The complexity of such trace analysis thereby depends on the accuracy 
and resolution of the deployed positioning systems. If the resolution and accu- 
racy is significantly better than the typical width of a lane (<0,5 m), the 
exchanged trajectories can be used to detect any location that has not been 
passed over by any vehicle for a certain period of time. However, the resolu- 
tion of currently deployed positioning systems is not yet sufficient enough. To 
overcome this problem, steering angles and speed reductions of the following 
vehicles can also be taken into account to further verify, if there is an unusual 
behavior of vehicles at (or close to) the reported location. 



OSGi Framework 
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Ad-Hoc Communication 
Service 


| j TCP/IP 




VehicleAPI Adapter 
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CAN - Bus 
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Fig. 7. System architecture 

An additional verification may be based on vehicles that are equipped with 
sophisticated sensor systems like radar. Today, these systems typically still 
suppress static objects because of performance reasons. Again, having evi- 
dence for an evasion maneuver could initiate a systematic evaluation of static 
objects only at that particular location. 
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Given sufficient evidence for an obstacle as described, a warning message can 
be sent to other vehicles whose drivers can be warned in time and are there- 
fore able to intervene adequately. 



5 Prototype 

For the purpose of verifying the operability of the proposed approach, we 
implemented the described algorithms into a BMW 5 series (E60) prototype 
vehicle. In addition, the system was linked to existing software modules that 
enable direct sensor access to the vehicle's CAN buses, and intervehicle com- 
munication based on IEEE 802. 1 la to populate warnings to other vehicles after 
an obstacle has been detected. 

In order to enable data access, a dedicated interface (VehicleAPI) was designed 
providing any requested data with sufficient quality of service (QoS). This 
interface is not addicted to any specific brand or model. 

The overall architecture is service oriented and based on the OSGi Framework 
(Fig. 7). Furthermore, a graphical interface was developed, where the car ride 
can be traced to check if data is interpreted correctly. 

One of the core aspects of our concept is the fact that the system has access to 
common sensor data, usually provided by in-vehicle bus systems like CAN. 
This enables third party solutions that can be offered in the aftermarket and 
therefore ensure a quick deployment. If offered in the aftermarket, the only 
customization that has to be done is the adaptation to the particular CAN for- 
mat of the respective vehicle. 



6 Evaluation 

In order to prove the developed concept, numerous test cases had to be estab- 
lished. To demonstrate the correctness of our approach, black box tests had 
been used, where primarily the overall functionality of the approach was test- 
ed. 

There are basically three different situations that have to be distinguished: an 
evasion manoeuvre, regardless of the dimension of an obstacle, hectic driving 
style and regular driving conditions. Each case has been tested several times 
with different drivers. After configuring some settings like the lateral acceler- 
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ation threshold and measurement time intervals, our planned results had been 
achieved strongly. Hence, our approach has been proved as a solid concept for 
obstacle detection without using additional complex hardware. 

Even though we have reached a high success rate, it can not be assumed as a 
system which provides 100% reliability in the current state. One conceivable 
and powerful add-on could be the availability of a digital map, which is sup- 
posed to ensure a better forecast of the reference route. Instead of the present 
lightweight route estimation, the real road geometry at a certain location 
could be used. 



7 Conclusion 

We have presented a new, cooperative and lightweight approach to deduce evi- 
dence for the existence of a road obstacle. The main idea is to systematically 
analyze the dynamic behaviour of vehicles. Therefore, no additional sophisti- 
cated sensor systems are necessary. Instead, we assume that vehicles are 
equipped with an ad-hoc communication system. Thus, vehicles are able to 
share individual knowledge among each other. 

In a first step, a potential evasion manoeuvre is deduced that indicates a poten- 
tial obstacle at this position. In order to exclude false alerts, a subsequent ver- 
ification process is started, exploiting the trajectories of the following vehicles. 

We implemented the proposed approach into a prototype vehicle and showed 
that it performs well under real world conditions. The proposed system is 
cheap and easy to integrate, and intended to complement sophisticated but 
expensive stand-alone sensor systems that aim to reduce the number of acci- 
dents through foresighted driving assistance. 

This work has been carried out in the context of the wireless local danger 
warning (WILLWARN) project within the PReVENT research programme, 
funded by the European Union. 
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Abstract 

Far Infrared (FIR) technology was originated a few decades ago in 
the military domain for its ability to detect any dissipation of heat, 
as this reveals not only the presence of moving threats or vehicles, 
but also of living creatures, including human beings, in a passive 
way, by daytime or night time, even in adverse conditions. More 
recently, FIR was found a valuable sensor to detect Vulnerable Road 
Users (VRU) in the automotive safety domain. After a first adapta- 
tion to VRU detection of typical FIR detection techniques used in the 
military domain, where most applications tend to rather concentrate 
on small objects at a great distance, we have also investigated more 
recent theoretical schemes, applying them to Far Infrared images. 
Candidate schemes include various learning methods, such as neural 
networks, support vector machine, boosting. 



1 Introduction 

Annual statistics recall the dramatic toll of road fatalities. As more advanced 
technology becomes available aboard vehicles with time, it seems only rea- 
sonable to use growing capabilities with utmost priority in order to help reduc- 
ing the number of victims. A number of EC-sponsored projects are setting 
ambitious goals for on-going research in the domain of active safety. The study 
described here was developed within one of these. Integrated Project 
PReVENT (www.ip-prevent.org), and more specifically for the COMPOSE sub- 
project. 

The "driver and vehicle" system is a complex one, which evolves within the 
road context and interacts with other road users, vehicles, pedestrians, etc. 
Accidentology shows that accidents can be explained by various sources of fail- 
ures of this system, at the level of the perception of the environment, or at the 
decision level, or implementing the action decided. These different failure lev- 
els are addressed by various research initiatives, detecting potential threats or 
conflicts, or conveying important information to the driver, or taking the cor- 
rect decision, or concretely triggering specific mechanical devices. Moreover, 
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some projects address more than one level of a given application, which can 
resort to intersection management, or longitudinal control etc. 

PReVENT/COMPOSE subproject focuses on pre-crash and collision mitigation. 
Therefore, in relevant situations, collision can no longer be avoided, but active 
measures can still decrease the severity of damages. The overall idea behind 
dedicated devices is to decrease the amount of cinematic energy at the time of 
impact when it involves Vulnerable Road Users (VRUs), as active bumpers do 
for example. Along with reducing the amount of energy, the geometry of such 
devices aims at reducing the severity of consequences, trying not to impact 
fragile and vital parts of the VRU's body. 

Deciding to trigger such a device is an important and often irreversible choice. 
Taking such an option must rely on a comprehensive perception of the overall 
environment and detection of VRUs. A main trend within the perception com- 
munity in various applications includes considering an array of different sen- 
sors to properly assess the environment. Long range radars, visible, near- 
infrared or far-infrared (FIR) cameras, short range radars, laser-scanners are 
only examples of such sensors being scrutinized. Managing information com- 
ing from different sensors implies architecture, communication, registration 
issues and fusion algorithms which are not considered here. 

We concentrate on FIR imagery, as this sensor proves to be very fit for VRU 
detection. After establishing the practical and geometrical context for taking 
images, we report different algorithmic approaches relevant for VRU detec- 
tion. 



2 Scope and Practical Context 

2.1 Perception for VRU Protection in Pre-Crash and Collision Mitigation 

The first point to set is the horizon monitored by the system. Pre-crash and col- 
lision mitigation are only concerned with objects within a short range, so close 
to the vehicle that collision can no longer be avoided. Practically, a range of 30 
meters seems the maximum distance to consider. 

Once range is set, the next key parameter is the field of view. We consider that 
selecting an horizontal field of view below 40° would result in too late a detec- 
tion of a VRU crossing toward the space soon to be filled by the vehicle. A ver- 
tical field of view of the same order is always sufficient. 
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Dist=30meters 




Fig. 1 . Monitoring geometry 

Once we know how far we should watch, the next question is which objects 
are relevant. Here, FIR technology was selected due to its capability to 
enhance VRU visibility. The order of magnitude of the height of a VRU will be 
roughly around 1.60 meter, not as an accurate dimension, but rather to prop- 
erly dimension image capture. 



2.2 Additional Constraints 

Since the price of a FIR camera is a key factor of the cost of a VRU protection 
system, and knowing that its cost depends mainly on image resolution, we 
have to consider using sufficient, but not oversized images. A common alter- 
native opposes 160x120 images to 320x240 images. 

Finally, the vehicle is already packed with many devices, and integration of the 
sensor aboard the vehicle can be a touchy subject. The most common location 
acceptable for a camera lies inside the front bumper. 



2.3 Image Capture Geometry 



The previous sections has set figures for perception geometry. 
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Dist=30meters 





Fig. 2. Image capture geometry 

Based on these assumptions, we can easily derive the number of pixels corre- 
sponding to the height of a VRU at the maximal relevant distance. 



VRUHx 



Npix / 



VRUhpix = - 



Dist x tan 



( fo %) 



(1) 



which yields approximately 23 pixels for the height of a pedestrian, assum- 
ing a 320x240 image. 

Fig. 3 is an example of a 32x16 pedestrian sub-image, extracted from a regu- 
lar image. Intuitively, we can assume that recognizing VRU using images with 
less pixels can hardly be thought of. However, the order of magnitude of the 
image dimension selected seems to make the task feasible, provided that ade- 
quate algorithms are implemented. 
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Fig. 3. Pedestrian image example 



3 General Properties of Detection-by-Learning Schemes 

3.1 Sub-image-based Recognition 

With increasing computing power, most advanced recognition techniques 
tend to turn from dedicated algorithms, which are fine-tuned to zoom on one 
given shape or very similar instances, to learning algorithms, which follow a 
general-purpose procedure to elaborate a basic recognition module based on a 
set of examples. The resulting module, given a candidate image, decides 
whether it represents an object of the category it was trained to detect. The 
object must basically fill the input image to make positive decision possible. 

Therefore, the whole problem of detecting VRU, as an example, can now be 
split in two parts: 

► Image scanning: selecting and/or building sub-images from the image 
delivered by the camera to present to the basic recognition module 

► The basic recognition module itself 

Both parts are of major importance, as they may influence performances, time 
response and required computing power with similar weights. 



3.2 Image Scanning Mechanism 

Scanning the image consists in extracting a number of sub-images from the 
camera image and feeding them to the basic recognition module. For pedes- 
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trians, for example, we might want to consider vertical rectangles (height = 
double of width). Assuming a given size for a pedestrian (like 32x16 pixels), we 
shall consider a whole grid of candidate positions for a pedestrian sub-image, 
as displayed below. Note that neighbour sub-images will generally largely 
overlap. 
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Fig. 4. Image scanning mechanism 

Moreover, pedestrians may appear in the image with different sizes, depend- 
ing on their real size and the distance to the camera, so we shall consider dif- 
ferent scales, i.e. different sub-image sizes, and in turn different grids for dif- 
ferent scales. Grids naturally become sparser as the sub-image grows. 
However, the total number of positions to consider for recognition includes all 
locations from all scales. This number can be a few 10 5 for a brute force scan- 
ning of a single image. This will results in a high number of candidates pre- 
sented to the basic recognition module, so in turn not only in heavy computa- 
tion, but also possibly many false positives, regardless of the performance rate 
of the basic recognition module. 



3.3 Sub-Image Positive Databases 

To elaborate an efficient learning-based recognition, the first step consists in 
gathering a great number of sub-images representing full frame candidate 
objects, VRUs in our case here. Here, we benefit from the relative invariance 
of infrared images, which remains largely unaffected by light changes, weath- 
er conditions, a.s.o. 
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Fig. 5. VRU sub-image examples 



This can be a tedious and lengthy phase, for which no fully automated tool 
exists. Therefore, we shall rely on observations made and recorded by an oper- 
ator pointing at pedestrians in existing images. Sub-images are systematically 
scaled to an appropriate standard size. 



3.4 Sub-Image Negative Databases 

Although this point might not be so intuitive, a learning algorithm also needs 
an important set of counter-examples on which it knows it should give a neg- 
ative answer. The size and pertinence of the negative database should not be 
underestimated. 

Practically, building a consistent negative database can be made much easier 
than it is to build its positive counterpart. Various procedures can deliver 
appropriate sub-images a priori not centred on VRUs, and it is enough to inval- 
idate any sub-image appearing to display a full frame VRU. 



4 Boosting Techniques 

4.1 Introduction to Learning Techniques 

Having earlier experience with using neural networks for classification, we 
have carried no further experiment on this approach. 

A survey of literature reveals many articles about pedestrian detection using 
image processing techniques. A short list of relevant references which were 
analysed and discussed is given in the "References" section, [1-14]. More ref- 
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erences are also given, pertaining to learning techniques in general, or applied 
to different applications. 

In any case, we shall always assume a training set, including positive and neg- 
ative examples - with/without the objects of interest - and a training phase 
using this set to tune the algorithm. Then the algorithm is tested and evaluat- 
ed on a test set, which does not overlap the training set. 



4.2 Boosting 

Boosting is a general technique applied in learning. It consists, during the 
learning phase of the algorithm, in reinforcing misclassified examples weights, 
so as to pay more attention to them during the following learning step. Fig. 6 
illustrates how boosting can converge toward a better solution. 



Boosted weights 




Fig. 6. Boosting 

Classifier # 1 has achieved a reasonable classification of positive and negative 
example, displayed by a line. Therefore, the three misclassified examples will 
be given increased importance during the next step. Not surprisingly, classifi- 
er # 2, selected based on updated weights, will classify these examples cor- 
rectly, leaving other examples out, which will in turned be given increased 
importance when selecting classifier # 3, which, combined with predecessors, 
will improve overall performance. 

Freund [17] proposes to build up a series of strong classifiers, each of them 
combining many weak classifiers. Strong classifiers are lined up in a serial 
way, so that the second strong classifier only knows candidates accepted by 
the first classifier, and so on. 
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Practically, a weak classifier can be based on thresholding some simple feature 
which can be computed from the candidate sub-image in a very short time. 
Learning means selecting a good list of classifiers from the acceptable diction- 
ary of features, with correct threshold. Tuning the threshold is illustrated in 
Fig. 7. 



Example database 
Negative, Positive 




Feature output on database 



Threshold 



Feature 
+ Threshold 
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Fig. 7. Weak classifier tuning 



Tuning the threshold of a classifier consists in selecting the value which best 
separates negative and positive feature values. 




Fig. 8. Strong classifier structure 

A strong classifier combines linearly binary output of weak classifier through 
weights, and compares the sum with a threshold. This structure allows boost- 
ing during the training phase. 

Strong classifiers are built up one by one, starting from the first step. In order 
to achieve a given false positive rate for the whole classifier, we may relax the 
specification for each single step. Once the first step meets this specification 
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on the whole initial database, the second step is trained to achieve the same 
performance, but only on the examples which were classified positively by the 
first classifier. Following steps are built in the same manner. 



Strong Classifier #2 
Candidates \ 



Accepted 






F 'F 



Rejected 



Fig. 9. Full classifier structure 



4.3 Application to Pedestrian Detection 

A comprehensive description of a training/recognition scheme has been given 
above. However, we still have to define the family of features to consider to 
build weak classifiers. Viola and Jones [26] suggest a family of simple inte- 
grating kernels, adding the pixel values where the shape is marked white, sub- 
tracting where it is marked black, ignoring other pixels. 






Fig. 10. Simple sub-image features 

Roughly, the features illustrated above aim at capturing successively edge 
information (first two templates: vertical edges, horizontal edges), line infor- 
mation (templates # 3 and 4: vertical lines, horizontal lines) and local minima 
and maxima (last template). These templates are considered for all sizes, 
which amount to a fair number of candidate features. 

Such features can be easily and quickly computed over a whole sub-image, and 
used during training phase to derive a cascade of strong classifiers as 
described in previous sections. 




[J H 





Far Infrared Detection Algorithms for Vulnerable Road Users Protection I 165 



4.4 Experiments and Results 

The technique described here is only a basic recognition module, which should 
be at least combined with a scanning mechanism. We give performance esti- 
mates of the basic recognition module, and translate it in a straightforward 
manner into a overall performance. 

The overall performance of such a basic recognition module, although reach- 
ing a detection rate around 85% on the training set, falls down to 50% on a dif- 
ferent database. The false positive rate is very low, but we should keep in mind 
that a great deal of candidates will be presented. These figures only represent 
the performance of the internal recognition module, not the performance of 
the FIR VRU detection. 

Building a very straightforward VRU detection system using only such a detec- 
tion technique would roughly achieve a 50% detection performance - equal to 
the basic recognition module performance - and give an average of 1 false pos- 
itive for each image processed. This may seem a poor performance, and clear- 
ly does not meet practical requirements. However, taken into account that 
such the basic recognition module processes a huge number of sub-images, its 
performance can be thought as perfectly fair. 

Many experiments were conducted, restricting or enlarging the family of fea- 
tures, or changing the algorithm at many levels, resulting in only slight gains. 

A major point of this recognition technique is the level of performance 
achieved with respect to the amount of computing required at detection time, 
which is very low. From this point of view, this technique seems to have little, 
if any, competitors. 

Finally, the positive conclusion of this study, is that the recognition module 
exhibits extremely interesting and relevant capabilities, but that building a 
complete VRU detection system requires integrating the combination with 
complementary modules as found in most - if not all - detection system: pre- 
processing , tracking, confirmation, etc. 
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5 Support Vector Machines 

5.1 SVM Principle 

Vapnik [25] introduced SVM classification, a technique supported by a solid 
theoretical background. Basically the descriptors are points of a high dimen- 
sion space. A nice property of such a space is that separation of positive and 
negative examples by a hyper-plane may be a likely hypothesis, which is high- 
ly unlikely in 2D/3D spaces with which we are naturally more familiar. 

Then the problem becomes: "How can we find a right hyper-plane to separate 
both sets?" 



Fig. 11. Support Vector Principle 

SVM defines the margin as a quality measurement of the hyper-plane. It can 
be understood that without a sufficient margin, the classifier might be 
extremely touchy, and it would become unlikely that any classification quality 
evaluated on the training set might still hold on a different testing set. 

An extensive literature on SVM (see [15], [16], [21], [22], [23], [24]) provides 
hints on how to represent examples, and how to compute a hyper-plane with 
good separation properties. In some cases [22], the whole processing algorithm 
boils down to solving a least squares linear problem. 
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5.2 Implementing SVM for Pedestrian Detection 

A straightforward implementation of SVM uses sub-images as input vectors 
and computes a separating hyper-plane, similarly to how boosting algorithms 
were implemented, using the benefit of infrared image invariance. As for any 
classification problem, the performances are evaluated in terms of detection 
rate and false positives. 

We tuned the parameters of the algorithms in an attempt to reach a decent - 
not too high - false positive rate. As a consequence, this translates into a lower 
detection rate. 



5.3 Results 

Learning performances obtained with SVM on our training sets appear some- 
what similar to what boosting algorithms achieved. However, the generaliza- 
tion capabilities, evaluated on different testing sets, seems lower. The general 
tendency in the results we obtained is a high false positive rate (>20 per 
image) when trying to achieve a decent detection rate (> 70% !) and a low 
detection rate when trying to contain the false positives. 

We have also shown that the full dimension of sub-images was not necessary 
to achieve the performances, which might open ways to further reduce com- 
putation effort. 

As a conclusion, our problem turns out to be too difficult to be solved single- 
handedly by such a SVM technique. 



6 Conclusion 

We have investigated different learning-based VRU detection algorithms on 
infrared images. The diversity of technical conclusions cannot be fully 
accounted here, but in a synthetic way, we found that boosting-based recogni- 
tion techniques, although not delivering the final performance that we need 
for a standalone system, exhibit excellent capabilities when compared to the 
amount of computation required. 

SVM techniques seems to give rather permissive performances, that is, giving 
a important number of false positives. Therefore, like boosting, SVM does not 
seem a sufficient tool to detect VRUs reliably. 
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However, both techniques have individual qualities that ought to be consid- 
ered, not using either as a full classifier - as was naively thought in the first 
place - but as modules of a complete classifying system, together with more 
classical blocks such as tracking a.s.o. 

This direction is currently under investigation. 
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Abstract 

In 2004 the NHTSA FMVSS-208 ruling became effective, to avoid 
injuries of children by unwanted deployment of the passenger 
airbag. To fulfil the requirements of this law, several technical prin- 
ciples are under development or had a first product launch recently 
The key technologies are weight sensing by measuring strain or dis- 
placement or mat solutions either fluid filled or as pattern recogni- 
tion. This paper presents the Bosch iBolt technology, a direct weight 
measuring system, which is an integrated and therefore functional 
part of a seat. The advantages and properties of this technology will 
be described depending on the main questions, like insensitivity 
against lateral forces and moments, temperature behaviour and 
packaging issues both on component and seat level. The weight 
sensing iBolts are installed directly into the seat frame, replacing 
existing bolts, so no H-point shift is caused by integrating the iBolts 
sensors. 



1 Introduction 

Since 2004 the NHTSA FMVSS-208 ruling (National Highway Traffic Safety 
Administration; Federal Motor Vehicle Safety Standards and Regulations 208) 
is effective. It was introduced for reduction or prevention of (fatal) injuries of 
children by unwanted deployment of the passenger airbag. In order to fulfil 
this regulation several technical solutions are being currently developed or 
have been launched recently. The working principles are based mainly on 
weight sensing utilizing strain or displacement measurement. Other systems 
are utilising a gel-filled or keypad-like mat-solution for pattern recognition. 
Every sensor system has to fulfil all the NHTSA FMVSS-208 rules. Under all 
circumstances the occupancy status of the passenger seat has to be deter- 
mined accurately. On top of the 208 requirement, customers demand field 
robustness over lifetime to prevent any misclassification. This indicates the 
exceedingly high requirement on the functionality of a standardized sensor 
system. At the same time the various environmental influences, such as floor 
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panel, seat type, seat cushion, seat cover, seat installation tolerances, etc. have 
to be considered. 

This paper addresses the BOSCH iBolt™ technology, a direct measuring sys- 
tem, directly integrated - and therefore a functional part - into the seat struc- 
ture. Properties, performance and advantages of the BOSCH iBolt™ technolo- 
gy are described. Various key areas, e.g. sensitivity against lateral forces and 
moments, temperature performance, hysteresis, packaging issues, are dis- 
cussed on both component and system level. 

The working principle of the BOSCH iBolt™ is based on measuring the deflec- 
tion of a bending beam. The deflection is monitored by measuring the change 
of static magnetic field utilising a specific Hall-sensor-to-magnet field arrange- 
ment. The BOSCH iBolt™ is designed in such way that the change of magnet- 
ic field is proportional to the load present on the seat. The unique design of the 
BOSCH iBolt™ enables an exceptional insensitivity against lateral forces and 
moments. Consequently the BOSCH iBolt™ is capable of measuring the correct 
load under a wide range of environmental strain conditions. This in return per- 
mits simple and inexpensive seat structure integration. Utilising a high 
strength steel in combination with a limit stop design ensures the outstanding 
accuracy and temperature behaviour of the BOSCH iBolt™. The limit stop 
design prevents yielding of the bending beam at low speed crashes. 

The particular design of the BOSCH iBolt™ allows for an assembly within a 
common seat structure simply replacing an existing bolt. Hence no H-point 
shift is caused by integrating the BOSCH iBolt™, which is a major advantage 
compared to systems mounted underneath the seat track. Additionally, since 
the BOSCH iBolt™ is integrated into the seat structure there are no effects of 
seat type or seat cushion on the iBolt™ performance. In case of implementing 
the iBolt™ above the seat belt attachment point a customized algorithm in the 
ECU usually handles correct occupancy detection without an additional belt 
tension sensor. Incorporating the BOSCH iBolt™ below the seat belt attach- 
ment point eliminates the necessity of an additional belt tension sensor. 



2 System Design 

The BOSCH iBolt™ system consists, depending on the application, out of up to 
four iBolt™ sensors integrated into the seat structure and an Electronic 
Control Unit (ECU). A customized algorithm stored in the ECU classifies the 
passenger according to the weight signals of the sensors. This classification is 
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used by the Airbag-ECU to enable or disable the firing of the passenger airbag. 
Fig. 1 shows a schematic system layout. 

The communication of the weight signals to the ECU is based on an analogue 
point-to-point connection to the ECU. Furthermore it enables a localization of 
each sensor in the seat. This is necessary for diagnosis and service purposes in 
case of exchanging a sensor. For diagnostic purposes the ECU enters a digital 
communication mode with the sensor. This enables the ECU to read out sev- 
eral parameters stored in the sensor. 
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Fig. 1 . System layout of the BOSCH iBolt™. 

The sensor design allows for calibration of sensitivity and offset on sensor level 
as well as a temperature calibration. Hence the weight signal generated by 
each iBolt™ is fully temperature compensated. 



The analogue output signals of the iBolt™ are converted into digital values. 
Next the ECU low pass filters the sensor signals and summarizes them to a 
weight signal. The weight of the seat itself, stored in a non-volatile memory is 
continuously subtracted from the actually measured weight signal. Primary 
part of the algorithm incorporated in the ECU is the classification of the pas- 
senger. The main focus is on the differentiation between a child and a small 
adult. The trigger area is only 6 kg whether a passenger seat is occupied by a 
small child in its child seat (6 year old ATD in the heaviest NHTSA Booster) or 
by a so called 5 %-female (NHTSA specified 5%-female). Software filter stages 
are used to ensure a stabilized classification particularly under transient or 
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dynamic events. Additionally diagnostic functionality is implemented to guar- 
antee correct classification of the system. 

The result of the classification is transferred to the ECU using either CAN or 
other OEM specific protocols. Depending on the detected class the airbag ECU 
enables or disables the airbag. Alternatively an adjusted level of airbag firing 
is possible. 



3 Working Principle 

3.1 The Sensing Principle 

One function of the iBolt™ is to transform the effective force caused by the 
weight of a passenger into an electrical signal. The automotive coordinate sys- 
tem defines the driving direction as X, the vertical and horizontal orientation 
as Z and Y, respectively. The iBolt™ combines a bending beam with the Hall 
measuring principle. It is designed in a way that preferably the Z-component 
of the weight of a passenger causes the displacement of the bending beam. 
The Hall-to-magnet arrangement within the iBolt™ is such that a linear 
change in the electrical signal is achieved. Thereby a static magnetic field, 
running through a Hall-sensor, causes an output signal linear to the deflection 
of the bending beam. The unique design of the iBolt™ inhibits a horizontal dis- 
placement of the magnet relative to the Hall. Additionally the iBolt™ features 
a mechanical limitation of the maximum strain on the bending beam. 





Bi-directional 
measurement: 
compression & 
tension 



Integrated 

overload 

protection 



F > F limit 

gap closed 



Fig. 2. Schematic of the measurement principle of the BOSCH iBolt™ 
technology. 



iBolt Technology - A Weight Sensing System for Advanced Passenger Safety I 1 75 



Gaining a linear signal has been obtained due to a constructive detail. The load 
caused by a passenger is led from the upper seat structure into the sleeve and 
further into the bending beam (Fig. 2). From the bending beam the force is led 
into the seat track of the car. The bending beam is constructed as a double 
bending beam. Typically a double beam shows an S-type deflection line. 
Thereby both vertical joints of the double bending beam remain in a vertical 
position for any deflection. This feature ensures a parallel movement of the 
magnet to the Hall. Hence a linear output signal is achieved (Fig. 3). 




Fig. 3. Typical output signal of a BOSCH iBolt™ sensor as a function of the 
applied force. 

Tests in passenger seats have shown that sensors are exposed to positive and 
negative forces. The origin of these forces is twofold. First, seat-inherit pre- 
loads are effected due to manufacturing and assembly tolerances as well as 
from fitting into the car. Secondly different load situations are caused by the 
seating position of the passenger and tilt of the backrest (Chapter 4 and 5). 
Therefore the iBolt™ is designed to measure loads in positive and negative Z- 
direction. This allows for an unambiguous interpretation of the weight of the 
passenger. Due to the symmetric design of the iBolt™, it measures pulling and 
pushing forces at equal magnitudes and tolerances. This unique feature allows 
for using the same iBolt™ weight sensor at the four linkages of upper seat 
structure and seat tracks in both vertical assembly directions. 



3.2 The Seat Linkage 

As mentioned earlier, the BOSCH iBolt™ has a second feeding. Like described 
above the accurate measurement of the weight of the passenger is only one 
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requirement of the system. Secondly the iBolt™ is designed as the linkage 
between upper seat structure and the seat tracks. It replaces an existing struc- 
tural part of the seat adopting an essential safety feature. In case of a crash the 
iBolt™ in conjunction with the seatbelt have to divert the occurring forces to 
the floor panel anchorage points. 

In practice two crash situations are looked at, first a low speed crash up to 12 
mph must not cause any change in performance of the iBolt™. Secondly at a 
high speed crash the iBolt™ must ensure the integrity of the seat structure. 
Therefore, to limit the maximum stress the bending beam experiences, a 
mechanical limit-stop design is implemented into the iBolt™ assembly. While 
exceeding the maximum load Flimit the front joint of the bending beam 
mechanically engages with the limit-stop sleeve. This causes a direct trans- 
mission of the load from the upper seat structure into the seat track by-pass- 
ing the bending beam (Fig. 2, right hand side). As a result the force-sensitive 
bending beam is protected against plastic deformation. The FMVSS-208 regu- 
lations do not require a determination of loads above a certain value. 
Furthermore the loads in the relevant area can be determined more accurate- 
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Fig. 4. Component performance after low speed crash separated in X- and 
Z-component. 



3.3 Component Performance 

Measurement results on component level have shown that there is a sound lin- 
ear relation between the forces applied and measured with the iBolt™. The 
output signal is linear within the guarantied measurement range of + F limit . 
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Above the mechanically limited measurement range the output signal 
describes a significantly reduced gradient. This occurs once the bending beam 
joint starts to engage with the limit-stop sleeve at higher loads. None the less 
it does not affect the accuracy of the iBolt™ at all. 

As described above requirements on the functionality of the iBoltTM are 
twofold. One is that the iBoltTM is not affected by a significant offset after a 
12 mph crash. Since the force of a frontal crash can be disaggregated in a force 
in X- and Z-direction the errors caused in X- and Z-direction are evaluated sep- 
arately. Forces occurring during a low speed crash are in the range of 6,800 N 
and 4,200 N in X-direction and in Z-direction, respectively. Fig. 4 shows the 
offset of the iBoltTM for a simulated low speed crash which does not exceed 
+ 1.3 N. The results are calculated from the original offset of the sensor com- 
pared to the offset after a test load is applied in the range of 1,000 to 10,000 N. 
The results shown in Fig. 4 demonstrate the outstanding performance of the 
BOSCH iBoltTM technology. 



4 Influences of Lateral Forces and Moments 

Measuring weight correctly at a seat environmental conditions set high 
demands for a sensor concerning insensitivity to lateral forces and moments. 
The seat tolerances are usually in the range of several millimetres. Therefore 
seats with such tolerances lead inevitably to high preloads (up to +300 N) in 
all possible directions. 

There are basically two options to handle such preloads in a weight sensing 
application: 

1 . Gimbal integration of the sensor in the seat. 

2. Designing the weight sensor towards insensitivity against any lateral 
forces or moments. 

Obviously the first option will introduce a high amount of additional costs and 
effort to integrate and decouple the sensor in the seat. However the BOSCH 
iBolt™ is particularly designed for insensitivity against any lateral forces or 
moments. Aiming for this option additionally eases the entire seat integration 
process. The existing joints usually only need moderate modification. Low sen- 
sitivity against lateral forces and moments is achieved by a special design of 
the bending beam together with an improved magnet-to-Hall-sensor arrange- 
ment. 
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Fig. 5. Definition of cross-axis loads. Moments are applied by offset loads 
in y-direction. 

In general preloads can be divided in forces and moments. Fig. 5 shows the 
definition of these loads for the iBolt™. F z is the only designated sensing direc- 
tion. Hence the iBolt™ is distinctively designed to minimize the influence of 
loads and moments adding unwanted signals to the output of the sensor. The 
actual sensitivities in F x - and redirection are established by applying a force 
(™600 N) to the sensor in the corresponding direction. The sensor output is 
referred to the applied load and calculated as percentage (Tab. 1). 

The moments M x or are engendered at the iBolt, by applying a force F y 
(™600 N) with a geometrical offset of 1 inch (0.0254 m) to the Y-axis of the sen- 
sor in Z- and X-direction, respectively. This method implies a combined load 
case of F y and the corresponding moment to the sensor. Hence, the sensor out- 
put is corrected by the sensitivity in Y-direction times the applied load in Y- 
direction. This corrected sensor output is referred to the applied moment 
(M ap p i = F y * 0.0254 m) to calculate the moment sensitivity of the iBolt™ in 
N/Nm. 

Tab. 1 shows the measured cross-sensitivities (F x , F y , M x , M z ). The results were 
measured at numerous samples. All measured cross-sensitivities are close to 
zero. This excellent behaviour of the iBolt™ under the influence of lateral 
forces and moments is reflected in the seat level measurements shown in 
chapter 5.2. 
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Cross Sensitivity 


Mean 


Sigma 


F x Sensitivity 


0.13 % 


0.08 % 


F y Sensitivity 


0.14% 


0.08 % 


M x Sensitivity 


0.41 N/Nm 


0.11 N/Nm 


M y Sensitivity 


0.06 N/Nm 


0.04 N/Nm 



Tab. 1 . Overview of Cross-Sensitivities for iBolt™. 



5 Seat Integration and Performance 

5.1 Requirements for Seat Integration 

As discussed earlier the major tasks for the sensor design is to ensure the seat 
integrity, even at a high speed crash, as well as accurately determine the actu- 
al load on the seat. Furthermore all the FMVSS-208 requirements have to be 
fulfilled. This in return affects the seat integration. The unique design of the 
BOSCH iBolt™ is an unified sensor system resulting in no H-point shift. 
Therefore no major design changes at vehicle level are necessary compared to 
systems integrated underneath the seat tracks. As mentioned previously the 
iBolt™ design shows a distinct insensitivity against lateral forces and 
moments. This results in an ample degree of freedom for designing the seat 
structure. 




Fig. 6. Example of the BOSCH iBolt™ integrated in a seat between seat 
track and upper seat structure. 
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The iBolt™ design features integrated overload protection as well as allowing 
for a flexible connector design. Fig. 6 shows an example of an integration of 
the BOSCH iBolt™ between the seat tracks and the upper seat structure. A 
CAD schematic is given in Fig. 7 showing the lever arm of the height adjuster 
from the upper seat structure which is secured with a clip ring. The second 
attachment point of the iBolt™ is the seat track featuring a poka-yoke design 
to ensure correct orientation of the sensor in the seat structure. The iBolt™ is 
fixed with a nut (M10) to the seat track. 




Louerseat track 



Fig. 7. CAD schematic of the BOSCH iBolt™ integration in a seat struc- 
ture. 

Essential for the performance of the iBolt™ arrangement is to avoid any load 
shunts by-passing the measuring system. Therefore the selected position in 
the seat structure, the seat structure design and the connector design has to 
ensure free movement between upper seat structure and the seat tracks for all 
seat positions. Integration of the iBolt™ into the seat structure below the seat 
belt attachment point eliminates the necessity of an additional belt tension 
sensor. However a customized algorithm in the ECU usually allows for an 
implementation of the iBolt™ above the seat belt attachment point still safe- 
guarding correct occupancy detection without an additional belt tension sen- 
sor. Beneficial is a seat structure design preventing shunting of any parts 
between sensor and floor panel. 



5.2 Seat-Level Performance of the BOSCH iBolt™ Technology 

In addition to component testing seat performance at system level is evaluat- 
ed. Measurements are carried out to reflect various mechanical tolerance influ- 
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ences, e.g. floor panel and seat track tolerances. Therefore a series like seat is 
fitted with four BOSCH iBolt™ sensors and an ECU. All shown measurements 
are carried out at room temperature (™25°C) with an unloaded (0 N) and a 
loaded (™400 N) seat. The measurements cover the entire range of seat posi- 
tions as shown in Fig. 10 and Fig. 11. Starting and end point is the so called 
design position where the seat is in a mid-mid position. This is not necessarily 
the geometrical middle position of seat height and seat track. Further seat 
positions are front (-f), middle (-m) and rear (-r) of the seat tracks. The seat 
height is varied between down (-d), middle (-m) and up (-up). The system is 
only once zeroed before a measurement cycle including all seat positions for 
the unloaded and loaded seat. For the system accuracy and hysteresis test the 
weights are placed on the seat as shown in Fig. 8. The seat is in design posi- 
tion and the weight is increased in steps of 200 N up to a maximum of 800 N. 
For all other measurements the weight is placed in the middle of the seat right 
in front of the back rest (dotted line). 



13 cm 26 cm 26 cm 13 cm 





= Weights 



Fig. 8. Schematic of the weight positions on the seat. 



Fig. 9 shows the applied load compared to the measured output signal of the 
system. The maximum deviation observed is 11 N is at a load of ~601 N. The 
maximum hysteresis (8 N) occurs at an applied load of 201 N. These output 
weights are computed by the ECU out of the signals of the four iBolt™ sensors. 
Note the seat used is a slightly modified series seat to accommodate the 
iBolt™ sensors. No additional adaptations are made to reduce seat-embedded 
preloads and moments. 
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Fig. 9. System accuracy and hysteresis test of a series like passenger seat 
at loads up to 800 N 




Fig. 10. System performance over entire seat positions at a variation of 
mechanical tolerance of e.g. floor panel at no additional load on the 
seat. 

Looking at the system performance for a variation of possible misalignments 
of the seat tracks on the floor panel the BOSCH iBolt™ technology shows out- 
standing performance. In Figs. 10 and 11 the deviation of the output signal of 
the iBolt™ is shown as function of different misalignments without and with 
an additional load of 398 N respectively. Within the figures the maximum 
error bands (±30 N) are given. In total six different misalignment situations 
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are investigated. Additionally a combination of two misalignments is tested. 
As a reference the design position is shown without any misalignments. The 
maximum deviation over all misalignments is 15 N and 22 N for the unloaded 
and loaded seat, respectively. This maximum occurs at a so called 4 mm snow- 
plow situation where both front attachment points are moved inward by 1 mm 
and both rear attachment points outward by 1 mm. Interestingly the combi- 
nation of snow-plow and one raised attachment point shows no further 
increase of deviation. Hence for a certain seat structure a maximum deviation 
is expected caused by the system-implicit preloads. 

Seat tests performed focus on the variation of the relative position of the four 
anchorage points of a passenger seat to each other. Therefore in one test one 
by one the four anchorage points are raised by 2 and 4 mm as well as lowered 
by 2 mm (-2 mm up). A further experiment varied the distance between the 
anchorage points 1 mm inside or outside. A variation of that test is the so 
called snow-plow situation where the front and the rear anchorage points are 
moved inside and outside, respectively. In the track alignment test inboard and 
outboard seat track are moved in X-direction by 2.5 mm to each other. The 
seat tracks also can be tilted inside or outside. Hence one or both seat tracks is 
tilted either inside or outside by 1 °. 




Track-m Track-r Track-f Track-f Track-f Track’in Track-m Track-r Track-r Repeal 



Fig. 11. System performance over entire seat positions at a variation of 
mechanical tolerance of e.g. floor panel at 398 N additional load on 
the seat. 

All these misalignment positions are thought of reflecting the tolerances of 
floor panel and assembling of the seat structure. Hence combinations of these 
misalignments can occur. This is accounted for in the combination of the snow- 
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plow and one raised attachment point. Results shown in Figs. 10 and 11 give 
the worst case scenario of the different misalignment tests. 

Further investigations included temperature cycling measurements on seat 
level with and without load. Fig. 12 shows the deviation of the sensor output 
related to the temperature. The temperature scale is on the right hand side the 
deviation is shown on the left hand scale. Interestingly the maximum 
observed temperature related deviation is 16 N and 13 N for the unloaded and 
loaded seat, respectively. This effect is addressed to the additional load of 
™ 400 N which is orientated opposite to some of the movement in the seat due 
to the temperature influence. 



Temperature Influence on System Output 
with ON of External Weight 




Deviation from Applied Weight Temperature 



Temperature Influence on System Output 
with 403N of External Weight 




Fig. 1 2 . Temperature behaviour of a unloaded (above) and ™ 400 N loaded 
seat (below). 
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FMVSS 208 requirements include 35 mph, 50% and 95% male ATD sled test 
on a rigid barrier according to US-NCAP and static anchor pull test (FMVSS 
207/210) with a maximum load of F max =19 kN. The BOSCH iBolt™ sensors 
passed all tests and maintained their structural integrity. 

Additionally 12 mph low speed non-fire frontal crash and 10 mph rear impact, 
50% male ATD, sled test with vehicle buck as well as 10 mph side impact (SID) 
according to FMVSS214 are passed without performance degradation. 



6 Summary and Outlook 

In order to fulfil the requirements of the FMVSS 208 Robert Bosch GmbH has 
developed a seat integrated weight sensor (iBolt™). A complete sensing sys- 
tem for weight classification consists of 4 iBolts™ and an ECU (Electronic 
Control Unit). The ECU filters the input signals of the iBolts™, computes the 
sum and delivers the occupancy status of the passenger seat. The occupancy 
classification is communicated to the airbag ECU. 

The measurement principle of the iBolt is based on a deflecting bending beam 
utilizing a particular magnet- to-Hall-IC configuration. 

The main advantages of the presented iBolt™ concept are: 

a. Low and high speed crash robustness: The iBolt™ is designed such that 
yielding or fracturing of the sensor is prevented by an integrated limit 
stop feature. 

b. Ease of application: The iBolt is easily integrated into a seat structure 
not causing any H-point shift, because it replaces an existing bolt of the 
seat structure. 

c. Low cross coupling: The iBolt™ design is optimized for insensitivity to 
lateral forces and moments. 

Overall this allows for an inexpensive and simple integration into the seat 
structure. Furthermore no additional measures are required at seat level to 
limit or reduce lateral forces and moments affecting the iBolt™. This insensi- 
tivity leads to an excellent seat level performance, even in heavily distorted 
seat structures. 

Additionally, the iBolt™ is optimized for temperature performance leading to 
minimal temperature dependency within the operating range of -40 °C to 
+ 85°C. 
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The next iBolt™ generation will feature a digital bus solution replacing the 
analogue point-to-point connection. Furthermore the integration of the iBolt™- 
ECU into the airbag ECU is in development, to reduce the overall system costs. 

Dr.-lng. Klaus Kasten 

PO-Box: 1342 
72703 Reutlingen 
Germany 

Klaus.Kasten2@de.bosch.com 

Keywords: weight sensing, occupant classification, FMVSS 208, Hall-sensor, force 

sensor, passenger airbag, iBolt 
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Object Classification exploiting High Level Maps of 
Intersections 

S. Wender, T. Weiss, K. C. J. Dietmayer, University of Ulm 
K. Ch. Furstenberg, IBEO Automobile Sensor GmbH 

Abstract 

An object classification system is introduced. The system observes 
the vehicle's environment with a laser scanner. Preprocessing and 
object tracking algorithms are applied. The object classification com- 
bines a pattern classifier with rule based a priori knowledge and 
high level map information. The pattern classifier uses significant 
features to calculate membership values for each class. These mem- 
bership values are verified and corrected by a priori knowledge. 
Furthermore, a precise position of the test vehicle is estimated. The 
positions of observed objects in the high level map can be deter- 
mined exploiting this information. As the object position is restrict- 
ed for some object classes, this knowledge can be used in the classi- 
fication, which significantly improves its performance. Finally, the 
system is evaluated with labeled test data of several sequences at 
different intersections. 



1 Introduction 

Nowadays, automotive research focuses on active safety systems. Since many 
applications need similar sensor data processing, a common algorithmic base 
for all applications can reduce the overall processing time. The output of these 
common algorithms can comprise a detailed vehicle environment description. 
This description should consist of position, size, orientation and velocity of 
observed objects. In addition, the type of the objects can be important. This 
information might be useful for several applications. Vulnerable road user pro- 
tection, for example, is based on the identification of pedestrians. Sometimes, 
also the identification of bikes is required to protect their drivers. Another 
example may be precrash applications. The airbag deployment can be con- 
trolled depending on the estimated weight of the type of the crash opponent. 
For this application, a separation of cars, trucks, small road users, and station- 
ary objects will be interesting. 



188 I Safety 



Especially the vehicle environment perception at intersections is an important 
and challenging task. It is important as a large amount of accidents occur at 
intersections and challenging as the road users move in multiple directions. It 
is not possible to restrict the orientation and the direction of motion as it is pos- 
sible on highways or motorways. Additionally a high traffic density may 
increase the complexity of intersection scenarios (Fig. 1). Therefore, present 
research concentrates on special algorithms for intersections [1]. 



Fig. 1. Complex situations at intersection scenarios: Environment model- 



Our approach distinguishes five types of objects. There are four moving object 
types: "pedestrian", "car", "truck" and "bike". Motorcycles and bicycles belong 
to the class "bike". The remaining class of stationary objects is called 
"unknown". The focus of this work is on the improvement of the object classi- 
fication at intersections. 



2 Sensor Configuration 

The algorithms use sensor data of a scanning laser range finder (laser scan- 
ner). This laser scanner is manufactured by IBEO AS, Germany. It is integrat- 
ed in the front bumper of the test vehicle (Fig. 2). Technical details of this sen- 
sor can be found in [2]. 



or 




ing algorithms need to handle high traffic density, multiple direc- 
tions of motion and partial occlusions 
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Fig. 2. Integrated laser scanner: The integration allows for an adaptation 
to the vehicles design. The sensor technology enables the usage 
under adverse weather conditions. 



3 Preprocessing and Tracking 

A segmentation algorithm clusters close distance measurements, which possi- 
bly belong to the same object [3]. The measurements are assigned to observed 
objects of former time steps. This object tracking is based on Kalman Filter 
estimation. Detailed tracking algorithms are described in [4]. 

In this work, all object models are free mass models. The reference point is the 
centre of gravity. An improved feature association is used. This feature associ- 
ation provides the possibility to assign more than one segment to an object. 
This was necessary, because objects can disintegrate in several segments in 
some cases. This disintegration is due to the distance based segmentation. If 
different parts of large objects are measured, the distance between these parts 
can be too large to assign all measurements to the same object (Fig. 3). 




Fig. 3. Object disintegration: Objects can sometimes disintegrate into sev- 
eral segments. If different parts of large objects are measured, the 
distance between these parts can be too large to assign all meas- 
urements to the same segment. 
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The association of all segments to the same object can improve the classifica- 
tion task, because the object size is a significant criterion for the object classi- 
fication. Especially cars and trucks can often only be distinguished by their 
size. The multiple segment assignment concentrates only on the classes 
"pedestrian", "bike", "car", and "truck". The identification of multiple seg- 
ments for background objects is quite difficult, because there are no assump- 
tions or restrictions for the shape of background objects. 

Two methods are applied to assign multiple segments to an object, which was 
already observed in former time steps. The first method uses the fact that dis- 
integration causes the appearance of new segments. Usually, all objects in the 
environment of these segments are already assigned to other segments. A seg- 
ment will now be assigned as additional segment, if several conditions are ful- 
filled. The additional segment must not be located at the boundary of the field 
of view. In this case, the observation of a new object is more probable. The 
resulting object must not be transparent. That means the laser scanner may 
not provide measurements behind the object (Fig. 4, left). The resulting object 
must be convex (Fig. 4, middle). This restriction is used, because pedestrians, 
bikes, cars, and trucks are usually convex. Finally, a smooth contour is expect- 
ed for these objects. Therefore, the corresponding endpoints of two assigned 
segments must be closer than all other pairs of points. (Fig. 4, right) 




Fig. 4. Criteria for the first method to assign additional segments: New 
segments will be assigned to objects with segments, if they cannot 
be assigned to an object without segments. The combined object 
measurements must not be transparent (left). They must be convex 
(center) and have a smooth contour (right). 
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The second method to associate additional segments is the usage of object 
speed. Moving vehicles must keep a safety distance to the next vehicle. This 
distance depends on the vehicle's speed. Drivers in Germany are already high 
fined, if they do not keep a safety distance of a quarter of the speed indicator's 
value in m. That means the safety distance d in meter can be calculated based 
on the object speed v in meter per second: 

d> v* 3.6s / 4 

Studies have shown that drivers do not keep even this minimal safety distance. 
Therefore, the algorithm uses a minimal distance of an eighth of the speed 
indicator's value in m: 



^min ^ 



v* 3.6s/ 8 



If the tracking algorithm can reliably estimate an object speed with former 
observations, the minimal safety distance will be used. All segments in front 
of or behind the object will be assigned to the object, if they are closer than 
c/ min . All assigned segments are now used to calculate a new value for the 
measured object size (Fig. 5). Afterwards, the method is applied again to 
assign additional segments. This loop is stopped as soon as the algorithm can 
not find any more new segments. 





Fig. 5. Second method to assign additional segments: If the tracking algo- 
rithm can reliably estimate an object speed, all segments will be 
assigned, which are closer than a safety distance in front of or 
behind the object. New measurements of the object's size will be 
calculated. 

Both methods add only segments of the actual time step. The calculated size 
is used as measured feature for the Kalman Filter, which also filters the object 
size. The association of multiple segments must be repeated in the next time 
step. 

Based on the tracking and preprocessing algorithms, significant features are 
extracted. These features are used by the classifier to estimate the object class. 
There are three types of object features. The first feature group is the contour 
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information. This group consists of features like width or length. The second 
group consists of dynamic features like speed, which are extracted from the 
object tracking. Furthermore, observed reflectors are used as features. The 
calculated features are described in detail in [5]. 



3 Classification 

3.1 Classification Overview 




Fig. 6. Classification system: A pattern classifier estimates a membership 
value for each class based on the features of the object. Rule based 
a priori knowledge is used to verify and correct the output of the 
pattern classifier. A temporal filter combines classification results 
of several time steps. The output class is selected by the maximum 
membership value. 

The object classification consists of four parts. The first part is the pattern clas- 
sification. The pattern classifier calculates a membership vector for each 
object based on the object features. This vector contains a membership value 
for each object class. The membership value 0 means, the object is improba- 
ble. The membership value 1 means, the object is very probable. 

The pattern classifier is based on neural networks, which provide low pro- 
cessing time and good classification results. The detailed pattern classifier 
architecture is described in [6]. The classifier estimates its internal parameters 
by a sample data based training procedure. The pattern classification is the 
application of the trained classifier to new sample data. 
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The second part comprises the verification and the correction of the class vec- 
tor. A priori knowledge is used to correct implausible membership values. 
Restrictions with respect to the objects size and speed are applied. The mem- 
bership values of implausible classes are set to zero. In addition, information 
about the location of observed objects in a high level map will be used to 
restrict the object class. The usage of maps is described in Section 5. 

The third part applies a temporal mean to the membership vector. This filter 
includes classification results of former time steps in the actual classification. 

Finally, the object's most likely class is chosen for the output by the maximum 
membership value. 



3.3 Sample Data 

The pattern classifier is trained with a set of training samples. This set consists 
of 38 sequences. These sequences provide laser scanner distance measure- 
ments of 10.000 time steps (10.000 scans). The system was evaluated with 12 
additional sequences, which consist of approximately 5.000 scans. These 
sequences are not part of the training data. 



4 Precise Ego-Localization using Precise Digital Maps 

Today's common digital navigation maps contain information about the traffic 
infrastructure, such as the number of lanes and their positions as well as spe- 
cial locations, such as hospitals, fuelling stations or restaurants. The position 
accuracy is in the meter level, which is sufficient for navigation purposes. This 
work uses more precise environmental description, which can be measured 
with sensors like laser scanners. In this section an approach for the precise 
localization of the test vehicle using landmarks is proposed. This information 
will be used in connection with precise digital high level maps to improve the 
object classification algorithms significantly. 



4.1 Determination of the Precise Position of Landmarks 

4.1.1 Grid Maps 

The position of landmarks used for the determination of the precise position 
and orientation of a vehicle in a specific street section is determined using grid 
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maps. Therefore, a mapping vehicle equipped with a laser scanner passes a 
street section. The distance profiles provided by the laser scanner are accu- 
mulated. The position of the vehicle during the mapping process must be 
known precisely to generate a consistent and accurate grid map. Therefore, a 
RTK-GPS (Real Time Kinematic) receiver in conjunction with an IMU (Inertial 
Measurement Unit) can be used. The accuracy of the position measurements 
of those systems is in the centimeter level. For limited regions up to about 80 
m the integrated wheel speed encoders and yaw rate sensor can be used to 
determine the position and orientation of the vehicle during the mapping 
process. In Fig. 7 a grid map of an intersection is shown. 




Fig. 7. Grid map of an intersection: The posts of the traffic lights are used 
as landmarks (features) for the ego localization approach. The WGS- 
84 coordinates of the features are shown. 



4.1.2 Landmarks 

Vertical elongated stationary objects, such as posts of traffic lights and traffic 
signs, which can be detected by a laser scanner, are used as landmarks (fea- 
tures) to determine the precise position and orientation of the vehicle. 

As mentioned above, the positions of these stationary objects are determined 
using grid maps. Therefore, a reference coordinate system for the features and 
the grid map is required. Most GPS receivers provide their position measure- 
ments in the geographic WGS-84 (World Geodic System) coordinate system. 
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Many common navigation databases based on GDF (Geographic Data File) also 
use the WGS-84 coordinate system. 

For this reason, the positions of the features are determined in the WGS-84 
coordinate system as the features can be related to GPS measurements and the 
features can be added to common map databases. For this purpose, a reference 
point of the grid map and its orientation to geographic north must be known. 
This reference point coincides with the starting point of the mapping process. 
The WGS-84 coordinates of this point can be determined using a precise RTK- 
GPS system providing position measurements in the cm-level. 

If RTK-GPS is not available, surveillance points, which can be found in urban 
areas in low distances, can be used to determine a reference point of a grid 
map. The determination of the WGS-84 coordinates of features is described in 
detail in [7]. In Fig. 7 extracted features and their WGS-84 coordinates are 
shown. These landmarks are added to a high level map. 



4.1.3 Ego-Localization 

As mentioned above, the position and orientation of the vehicle is determined 
using features. In the first step the features are transformed into the laser 
scanner coordinate system with respect to the position measurements of a 
standard GPS receiver. These standard receivers are already integrated in 
modern cars for navigation purposes. As the accuracy of the GPS position 
measurements are in the meter level, the transformed landmarks are trans- 
lated and rotated in the scan of the laser scanner as shown in Fig. 8. The main 
challenge is to associate the features to the distance measurements of the laser 
scanner as the environment is rapidly changing and features are often occlud- 
ed in urban areas. 

In order to solve the association problem a translation and rotation invariant 
representation of geometric arrangements was defined. In the first step the 
scan is clustered as described in section 3. Several segments are combined to 
a mesh. The mesh is then represented by a set of parameters which define the 
geometric constellation of the segments. However, the representation is posi- 
tion and rotation invariant. This set of parameters is then compared to a mesh 
constructed from features of the high level map. A new algorithm TrAss 
(Triangle Association) finds the optimal association between a segment-mesh 
and a feature-mesh. This algorithm allows the detection of visible features in 
the scan in one time step. After an established association, the feature shapes 
are matched with the shape of the segments in order to determine the exact 
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position of the features in the laser scan. Fig. 8 shows an example with real 
data. The localization algorithm is described in detail in [8]. 




Fig. 8. Left: The features were transformed into the laser scanner coordi- 
nate system with respect to the position measurements of a stan- 
dard DGPS receiver providing position measurements with an 
accuracy of several meters. Right: The segments of the clustered 
scan are associated to the transformed features using the TrAss 
(Triangle Association) algorithm. This association is visualized by 
the green lines. The successful association allows a correction of 
the test vehicle's position and orientation. 



4.2 High Level Maps 

The infrastructure is represented by precise high level maps. These maps con- 
tain information about the topology of traffic lanes and other static objects like 
traffic lights or traffic signs. In addition, they can store attributes, like the driv- 
ing direction of a traffic lane. 

The high level maps can be stored in the GDF format, which is used by cur- 
rent navigation systems. Compared to existing navigation maps, the position 
of the traffic lanes is measured more precisely. It is possible to convert the lane 
information to polygons, which describe the traffic lanes. The combined poly- 
gons form the road area of the intersection (Fig. 9). 
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Fig. 9. All traffic lanes of the high level map are combined to a road mask. 

The information about the road is used in the rule based part of the classifica- 
tion. The precise position of the test vehicle can be used to determine a pre- 
cise position of observed objects in the high level map. It is assumed, that cars 
or trucks can not appear beside the road. Thus, the membership value of the 
classes "car" and "truck" will be set to zero, if an object is located beside the 
road. Background objects are not assumed to appear on the road. Therefore, 
the membership value of the class "unknown" is set to zero for all objects, 
which are located on the road. These rules allow a classification of objects by 
their position even if the objects' contour is identical (Fig. 10). 



Unknown 

L 



Big 



Car, Truck 



Test vehicle with 
‘ Laserscanner 



Fig. 10. The position of measured objects can be used for classification. 

Objects, which are observed in areas, which certainly belong to the 
road, cannot be stationary objects. Cars and trucks cannot appear 
in areas, which certainly do not belong to the road. 



Furthermore, high level maps can contain areas, which are not of interest for 
the tracking and classification algorithms. These areas can be houses or gar- 
dens, for example. The areas should be ignored by all algorithms (Fig. 11). This 
is realized by removing all distance measurements of the laser scanner, which 
are located in such areas. The remaining area is the region of interest (ROI). 
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road 



Fig. 1 1 . Some areas at intersection such as houses or gardens may be not of 
interest for tracking and classification algorithms. These areas can 
be stored in the high level map and all distance measurements 
inside these areas will be ignored. 



4.3 Application of Uncertainties 

The test vehicle's position is calculated with a position uncertainty and an ori- 
entation uncertainty. Applying these uncertainties to the high level map estab- 
lishes several areas of confidence. The first area is the area, which certainly 
belongs to the road. The second one is the area, which might belong to the 
road. The remaining area certainly does not belong to the road. The uncer- 
tainties are also applied to the areas, which should be ignored. The areas, 
which certainly should be ignored by tracking and classification, are calculat- 
ed. 

The test vehicle's position uncertainty reduces the area, which certainly 
belongs to the road. At the same time, the area, which might belong to the 
road, is increased with respect to the road mask. The remaining area does cer- 
tainly not belong to the road (Fig. 12). 




Fig. 12. The uncertainty of the test vehicles position reduces the area, 
which certainly belongs to the road, while increasing the area, 
which might belong to the road. 
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The uncertainty of the ego vehicle's orientation also reduces the area, which 
certainly belongs to the road. The area, which might belong to the road, is 
increased. When rotating the mask by this uncertainty to the left and the 
right, the whole covered area might belong to the road. The area, which cer- 
tainly belongs to the road, must be covered by the rotated mask at all possible 
rotation positions (Fig. 13). When the mask is rotated, the points of the poly- 
gon move on a circular arc. This arc is approximated as a straight line (high- 
lighted in red), because the rotation is performed with a small angle. 




position 



Might belong 
to the road 




Certainly belongs 
to the road 



Does certainly not 
belong to the road 



Fig. 13. The uncertainty of the ego vehicle's orientation can be taken into 
consideration by rotating the road mask. Every point, which is cov- 
ered at least once, might belong to the road. The area, which is cov- 
ered by all rotated masks, certainly belongs to the road. The 
remaining area certainly d oes not belong to the road. 



The translative and rotational uncertainties are combined to calculate the 
areas of confidence. First, the road mask is enlarged by the position uncer- 
tainty. Then it is rotated by the orientation uncertainty to the left and o the 
right. The two increased road masks with maximum rotation are used to 
determine the areas. The corresponding polygon points are connected with 
lines. The whole area, which is covered by this rotation, might belong to the 
road. The area is calculated by starting at the point with the minimal x value. 
The algorithm successively chooses polygon lines of the outer contour of the 
structure (Fig. 14, left). 

The area, which certainly belongs to the road, must be covered by both rotat- 
ed decreased masks. A point of this area must be inside the first mask and stay 
inside the other mask during the whole rotation. This fact is used to find a 
starting point for the area estimation. A polygon point of one mask is chosen. 
The connection to the corresponding point of the other mask must completely 
be inside this other mask. Again, the algorithm successively chooses lines of 
the inner contour (Fig. 14, right). 
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Fig. 14. The area, which might belong to the road, is calculated based on 
two increased road masks, which are rotated by the maximum 
angle to the left and the right. The corresponding points are con- 
nected with lines. The outer contour is extracted. The area, which 
certainly belongs to the road, is calculated with two decreased 
masks with maximum rotation. An inner contour is extracted. 



The uncertainties are also applied to all stored uninteresting areas. The areas, 
which should certainly be ignored, are estimated. This is similar to the calcu- 
lation of the area, which certainly belongs to the road. All calculated areas are 
shown in Fig. 15. 
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belong to the road 




Might belong 
to the road 



Certainly belongs 
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Fig. 15. High level map with applied uncertainties: Several areas of confi- 
dence are calculated by applying the test vehicle's position and ori- 
entation uncertainties to the high level map. 



5 System Evaluation 

The system was evaluated with test data of several sequences at different 
intersections. Of course, the test data was not used to train the classifier. Two 
measures were calculated to analyze the system performance. The detection 
rate describes the percentage of objects, which are correctly detected and clas- 
sified: 
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7 . true positives 

detection rate = 

true positives + false negatives 



True positives are objects of the classes "pedestrian", "bike", "car", and "truck", 
which are classified correctly. This number is divided by the number of all 
objects of these classes. 

The false detection rate is the ratio of false detections to all detected objects of 
the classes "pedestrian", "bike", "car" and "truck" 

r , , . false positives 

false detection rate = 

false positives + true positives 

In this case a false positive can be a misclassification of a background object, 
but also misclassifications of other objects as "pedestrian", "bike", "car" or 
"truck" will be false positives. That means that even a car, which is classified 
as "truck", is a false positive. 




— Witti hig h <eve< 
mgps 

— — — Without hif^i level 
maps 



Fig. 16. System evaluation: The system performance is evaluated based on 
complex intersection scenarios. The operating point curves show 
the possible combinations of detection rate and false detection rate. 

The dashed line shows the performance without high level maps. 

The improved results of the classification with high level maps are 
shown by the solid line. 

Different possible combinations of the described measures are shown by an 
operating point curve in Fig. 16. The solid line shows the performance with 
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high level maps at complex intersection scenarios. The dashed line shows the 
performance without high level maps for the same scenarios. 

The algorithms are implemented on an Intel Pentium M with 1.7 GHz. The 
processing time for one time step is 17 ms. As the used sensor sample rates 
are 10 Hz and 20 Hz, the system is applicable in real time. 



6 Conclusions 

An object classification system was described, which combines pattern classi- 
fication and a priori knowledge. Significant improvements in the classification 
performance were achieved by the usage of high level maps of intersections. 
Localizing the road users with respect to a high level map enables the appli- 
cation of object class depending constraints. Additionally the preprocessing 
could be improved by the assignment of multiple segments to one object in 
case of object disintegration. The complete system is designed to be applicable 
in real time. 

Current work aims at an extension of the scenarios. The presented results 
show the system performance for complex intersection scenarios. However, it 
is to be expected that other a priori knowledge of scenario environments, such 
as "driving on a highway", can enhance the performance of the object classi- 
fication. Further improvements may be achieved by adapting the system to 
the requirements of a specific application. 
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Abstract 

A variety of safety-enhancing automobile features can be enabled by 
microsystems that can sense and analyze the dynamic 3D environ- 
ment inside and outside the vehicle. It is desirable to directly sense 
the 3D shape of the scene, since the appearance of objects in a 2D 
image is confounded by illumination conditions, surface materials, 
and object orientation. To overcome the disadvantages of 3D sensing 
methods such as stereovision, radar, ultrasound, or scanning 
LADAR, we present Electronic Perception Technology, an advanced 
range camera module based on measuring the time delay of modu- 
lated infrared light from an active emitter, using a single detector 
chip fabricated on standard CMOS process. This paper overviews 
several safety applications and their sensor performance require- 
ments, describes the principles of operation of the range camera, 
and characterizes its performance as configured for airbag deploy- 
ment occupant sensing and backup obstacle warning applications. 



1 Introduction 

A variety of safety-enhancing automobile features can be enabled by microsys- 
tems that can sense and analyze the dynamic 3D environment inside and out- 
side the vehicle. Safety features may include collision warning and avoidance, 
smart airbag deployment, obstacle detection such as backup warning, and 
parking assistance. Common to these applications is the need to detect, isolate, 
measure, locate, recognize, and track objects such as people, traffic, and road- 
side features. 

It is often proposed to perform these tasks using conventional 2D imaging sen- 
sors and analysis software, but achieving cost-effective and reliable perform- 
ance during all vehicular usage scenarios is a formidable challenge. The 
appearance of objects in a 2D image varies greatly, depending on illumination 
conditions, surface materials, and object orientation. These variations in the 
image complicate the task of software that must interpret the scene. On the 
other hand, the 3D shape of objects is invariant to those confounding effects. 
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Stereovision based 3D recovery is computationally complex and fails on un- 
patterned surfaces. RADAR, ultrasonic, scanning LADAR, and other ranging 
technologies are similarly proposed, but they have difficulty discriminating 
objects due to limited temporal or angular resolution; moreover, the need for 
specialized sensors for each safety function poses system integration chal- 
lenges. A single high frame rate focal-plane-array 3D sensor is desirable 
because it can serve multiple safety and convenience functions simultaneous- 
ly, allowing applications to jointly exploit shape and appearance information in 
a dynamic scene. The output of the sensor should be a sequence of 2D arrays 
of pixel values, where each pixel value describes the brightness and Cartesian 
X, Y, Z coordinates of a 3D point on the surface of the scene. 

Canesta has developed Electronic Perception Technology (EPT), an advanced 
range camera module based on pixels that measure the time delay of modu- 
lated infrared light from an active emitter. Electronic Perception technology 
permits machines, consumer and electronic devices, or virtually any other 
class of modern product to perceive and react to objects and individuals in the 
nearby environment in real time, particularly through the medium of "sight," 
utilizing low-cost, high-performance, embedded sensors and software. 

The camera module contains a light source constructed from a bank of 
infrared LEDs (or laser diodes), a lens system for the detector chip, a detector 
chip with 160x120 phase-sensitive pixels fabricated on standard CMOS 
process, and an embedded CPU for application processing. 

This paper first surveys several automotive safety applications. Next, quanti- 
tative performance metrics are defined for range cameras, which are neces- 
sarily evaluated on a different basis from 2D cameras. It explains geometric 
uncertainty, accuracy, and dynamic range as important factors in the per- 
formance of range sensors. Then it describes how the automotive applications 
requirements map to the sensor performance metrics. Next, the paper 
describes the principles of operation of the EPT range camera, highlighting 
design features that overcome challenges to this sensing method, including 
resilience to full-sunlight illumination. Finally, the paper presents two applica- 
tion-specific configurations of the range camera and characterizes its per- 
formance for those applications. 



2 Safety Applications and Sensor Requirements 

Growing government legislation, increasing liability concerns, and the 
inevitable consumer desire for improved safety make the introduction of new 



Performance of a Time-of- Flight Range Camera for Intelligent Vehicle Safety Applications I 207 



safety features a high priority for automakers. Today various sensing tech- 
nologies play a key role in delivering these features, detecting conditions both 
inside and outside of the vehicle - in applications like parking assistance, adap- 
tive cruise control, and pre-crash collision mitigation. Each of these applica- 
tions is characterized by a unique customized technology (e.g. ultrasonic, 
RADAR, LADAR, digital image sensing, etc.), which generally provides either 
a ranging function or an object recognition function. 

The need for investment in multiple disparate technologies makes it challeng- 
ing to deploy individual safety features as quickly or as broadly as desired. 

Future applications pose even more difficulties, as multiple features must be 
provided in a single vehicle. Plus, virtually all of the new sensing applications 
on automakers' roadmaps (e.g. pedestrian detection being planned in Europe 
and Japan) require both ranging and object recognition functions. Combining 
two incongruent technologies to accomplish this task (such as RADAR and dig- 
ital image sensing) is expensive, difficult to implement, and poses the addi- 
tional problem of inefficient development. 

The use of vision gives added levels of discernment to the air bag systems by 
providing static or dynamic occupant classification and position sensing. 
Further, the addition of a vision system inside the cabin enables other value- 
added applications such as abandoned baby/pet detection, personalization, and 
security. Applications for vision-based sensing outside the car are blind spot 
detection, vehicle lane departure, safety in rear vision, proximity of other vehi- 
cles around the vehicle, and off road and heavy equipment proximity sensing. 
The benefits of vision sensors are two fold. They provide enhanced visual feed- 
back to assist the driver in operating the vehicle. But more importantly, when 
vision sensors also provide range data, they provide the necessary information 
for advanced algorithms to achieve higher level of discernment and more accu- 
rate analysis of object motion dynamic. With such sensors, for instance, the 
system can use the shape differences between a person and a large box sitting 
in the front seat to deploy the air bag or not. 

We consider the requirements of some of these applications after defining the 
relevant range sensor performance metrics. 



2.1 Range Sensor Performance Metrics 

An imaging sensor's lateral resolution can be measured in pixels per degree 
(pixels/ 0 ), but field distortion in typical lenses tends to reduce the resolution in 
the periphery of the image. 
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The geometric measurement performance of a range camera can be charac- 
terized by several statistics. For any quantity repeatedly measured over time, 
we define accuracy as the RMS error between the time average of that meas- 
urement and ground truth, uncertainty as the RMS error between instanta- 
neous measurements and the time average, and total uncertainty as the 3- 
sigma confidence bound between the instantaneous measurements and 
ground truth. The spatiotemporal extent of the averaging regions for these sta- 
tistics should be chosen based on the application. For occupant sensing, head- 
sized regions spanning 20 cm x 20 cm over 40 frames are appropriate. 

Specifically, the range camera produces a 3D measurement P = (X, Y, Z) for 
each image pixel / at each frame t. By convention, the coordinate system is 
defined with respect to the camera, with the Z axis in the viewing direction, 
and X, Y being lateral directions. If Z’ denotes the time average of Z measure- 
ments, and Z* denotes the ground truth, then we may compute the statistics 
as follows: 

► Accuracy in Z is the RMS error 

v^j n i ( Z\ - Z*i)2 / N. 

► Uncertainty in Z is the RMS error 

V^ t (Z it -Z’)2/A/. 

► Total uncertainty is 

\jj,\ +3cr, 

where fi = I t ( Z it - Z*j) / N and a = I t ( Z it - Z k i - /i) 2 / N. 
Performance statistics for X and Y are analogous. 

For a conventional imaging camera, dynamic range is defined as the ratio of 
the largest non-saturating (# max ) input signal to the smallest detectable input 
signal (/ min ), and is reported as 20 log 10 (/ ma x/ # *min) dB. The objective of a high 
dynamic range imaging camera is to reproduce brightest and darkest seg- 
ments of the scene. The ambient light works in favor of a conventional camera 
as it illuminates the scene for better imaging. On the other hand, the goal of a 
range camera is different. A range camera is evaluated based on its ability to 
detect range regardless of the ambient light condition. Therefore, the dynam- 
ic range of a range camera is defined as the ratio of the largest non-saturating 
input signal for detecting a range to the smallest detectable range signal. The 
dynamic range of a range camera can be further refined as the value of the 
usable dynamic range that satisfies certain Z uncertainty requirements. 

Ambient illumination from the sun and artificial lights is pervasive in automo- 
tive applications. A critical requirement, therefore, is for the sensor to meet 
performance specifications even when viewing sunlit surfaces. Indeed, the sun 
at noon will project roughly 1 kW/m 2 on sunlit surfaces. The EPT camera mod- 
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ule's optics contains an IR filter to pass only light whose wavelength matches 
the active emitter (e.g. 870 nm), reducing the amount of surface sunlight that 
reaches the sensor to about 50 W/m 2 . Still, compared to roughly 2 W/m 2 pro- 
jected by the active light source of the camera, a significant amount of sun- 
light reaches the sensor. The range camera must provide the technology to 
cancel the effect of the remaining ambient light that reaches the sensor. 

Another challenge in automotive environments is operational temperature 
range, at least -40 °C to +85°C. The sensor performance must remain within 
the specification required by each application for the range of automotive tem- 
peratures. Finally, for exterior applications, the sensing modality must be 
robust to degraded viewing conditions of rain, mud and fog. The mud and rain 
effect can be mitigated by either placing the sensor inside the car (e.g. behind 
the window glass), or by carefully protecting the lens by designing appropri- 
ate enclosures, or providing a washing and cleaning system. 



2.2 Occupant Sensing for Advanced Airbag Deployment 

The functionality of high volume production air bag systems was once limited 
to detecting a collision and then deploying the air bag. Improper seating of 
children in the front passenger seat, small stature adults, and out-of-position 
occupants were therefore major considerations. Of concern is the firing of an 
air bag when a person's head is too close to any of the vehicle air bags or there 
is something blocking the effective deployment of the air bag. The critical out- 
of-position (COOP) zone near the dashboard is an area where a person is likely 
to be injured by airbag deployment. 

Government and Consumers are pushing the Automotive Industry for 
improvements to the air bag systems in vehicles on the road today. The U.S 
Department of Transportation's (USDOT) National Highway Traffic Safety 
Administration (NHTSA) has a legislative mandate under Code of Federal 
Regulations (CFR) Title 49, Chapter 301, Motor Vehicle Safety, to issue Federal 
Motor Vehicle Safety Standards (FMVSS) and Regulations to which manufac- 
turers of motor vehicle and equipment items must conform and certify com- 
pliance. Occupant crash protection (571.208) lists the requirements for the air 
bag systems [Dot04]. This document is the basis for the vehicle manufactures 
to improve the air bag systems. 

Advanced air bag systems are requiring additional input information. A smart 
sensor system would be desired that can discern presence, size, location with- 
in the passenger space, and category of object. Through the real time moni- 
toring of the space, the air bag system will have the ability of making more 
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informed decisions on when, if, and with how much force the air bag will 
deploy. An air bag system with smart sensors has also the potential of warn- 
ing the passenger of unsafe seating posture during the normal operation of the 
vehicle. 

Today's "advanced airbag systems" are primarily based on weight sensors 
located in the seat cushion or floor. However, weight sensors alone cannot pro- 
vide all the necessary information for intelligent deployment and furthermore 
pose implementation challenges for the automaker. They are difficult to inte- 
grate with the vehicle, e.g. requiring designing the seat around the sensor, and 
are different for each vehicle model, contrary to a common components strat- 
egy across the product line. 

In contrast. Electronic Perception Technology is ideally suited for passenger 
localization [Gok04a] and classification [Gok05]. The EPT range camera upda- 
tes three dimensional data to the air bag system every 16 msec. This gives the 
air bag system time to calculate the passenger's velocity and distance to the 
COOP boundary. With this information the air bag system can determine the 
air bag deployment strategy that is required by the situation, based on FMVSS 
requirements. 

For the front passenger occupant sensing application, the volume that should 
be monitored is the space from floor to roof, from seatback to dashboard. With 
the range camera mounted in the center of the headliner, the required field of 
view is typically 110° vertical x 80° horizontal, and the farthest object may be 
up to 100 cm from the sensor. Within this volume, the geometric uncertainty 
should be 0.5 cm or better, and commensurately the image lateral resolution 
should be 4 pixels/ 0 . The geometric accuracy should be 2 cm or better in the 
region likely to be occupied by the passenger's head. These specifications are 
needed by intelligent algorithms that predict the location of the head from the 
dashboard after a crash is detected. Furthermore, the distance of head from 
the side window pillars is useful for the decision on the deployment of the side 
air bags. The range camera should provide 3D measurements at least 60 times 
per second, with latency not exceeding 16 ms, to give the airbag system suf- 
ficient time to calculate the person's velocity and distance to critical zone 
boundaries. Lower latency values are possible by reducing the window of 
interest size of the camera frame and obtain a rapid burst of data for a subset 
of field of view. 
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2.3 Backup Obstacle Warning 

The current trends in car styling and vehicle configuration, as in SUVs and 
Minivans, have reduced backup visibility of the driver. Video and ultra-sound 
backup warning systems are now fairly common in vehicles. A backup system 
provides visual and/or audio feedback to the driver when the object appears 
too close in the trajectory of the car. A backup system can warn the driver if 
an object is in the blind spot of the driver, or assist the driver during the park- 
ing. Backup systems are presently implemented by placing multiple ultra- 
sound sensors in the bumper of a car. 

For the backup sensor to be useful for the driver, it has to correctly estimate 
the distance of the car to an object that is in the trajectory of the car, even if 
close to the ground. It should not be triggered by roadside objects or the road 
itself. Otherwise, the sensor can become a nuisance rather than being a useful 
tool for the driver. The 2D video sensors have the advantage of giving a good 
view of the back of the car. However, estimating the real-world distances from 
a video image is a difficult task for the driver. The 3D image sensors have the 
advantage of providing both the visual and positional information in this appli- 
cation. 

It is estimated that 90% of backup collisions with pedestrians occurred at a 
vehicle velocity of 8 km/h or more. At that speed, it is necessary to monitor 4 
m behind the car just to achieve 95% successful accident aversion [Gla05]. 
Accordingly, for the backup warning system, the volume that should be mon- 
itored is a 4 m wide lane from the back of car to 10 m behind the car, from 
ground to 1 m. The geometric accuracy should be 10% of the radial distance 
or better and the uncertainty should be 1.5% of the radial distance or better. 
This suffices to detect a 15 cm thick person or object lying on the ground at 4 
m distance. The image lateral resolution should be 4 pixels/ 0 in order to detect 
a standing person at 10 m range. Per ISO recommendation, the backup detec- 
tion system including range camera should provide an audible alert to the 
driver within 350 ms of the manifestation of a hazard. 



3 Principles of Operation 

A time-of-flight sensor module measures distance by observing the time delay 
AT between emission and detection of light that travels from an active light 
source, bounces off a surface in the scene, and returns to the camera. For a 
light source modulated at frequency f and surface at radial distance r, the 
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delay can be expressed as a phase 0 = 4 jtrf / c; hence, distance can be recov- 
ered by measuring phase. 

This section describes the principles of operation of the EPT range camera, 
highlighting design features to overcome challenges to this sensing modality, 
including resilience to full-sunlight illumination [Gok04b]. First we describe 
the operation of the detector pixels in the sensor chip, and then we detail the 
calculations performed by the CPU to recover phase and 3D coordinates. 



3.1 Detector Pixels 

The light source in a typical EPT range camera module is a bank of IR LEDs at 
870 nm wavelengths, switched on and off with 50% duty cycle at a frequency 
on the order of 44 MHz. Each pixel in the detector array contains a pair of 
charge accumulating gates, called A and B, which are alternately enabled in- 
phase and out-of-phase with the light source. The gates are reset to 0 at the 
start of each frame and integrate over the frame exposure time S, typically 1 
to 32 ms. As the surface distance and phase of received light varies, the 
amount of integrated charge in gates A and B will vary (Fig. 1). 

Even in the absence of modulated light, the charge in A and B would be biased 
from 0 due to dark current and ambient scene illumination. The offset is inde- 
pendent of the received phase, and the two pixels are enabled for the same 
amount of time, so the biases could be canceled by measuring only the differ- 
ential signal D = A - B. The latter signal is digitized by an ADC and stored in 
a frame memory accessible to the CPU. 

While ambient illumination does not theoretically affect the D signal, it could 
potentially cause the gates A and B to saturate. To prevent this effect, the sen- 
sor implements a patented Common Mode Reset (CMR) process that removes 
the same amount of charge from A and B gates, leaving the differential signal 
unchanged. Typically, 20 resets are applied during each exposure. The camera 
detector has also the ability to prevent the saturation of the differential signal. 
These measures to avoid saturation of the detectors are among key innovations 
that enable the EPT range camera to operate in high ambient illumination 
conditions typical of automotive applications. 
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3.2 Recovery of Phase and Scene Geometry 

While D depends on 0, phase cannot be determined from a single observation 
of D. Accordingly, the same process is repeated, sequentially or in parallel, 
wherein the light source modulation is delayed by 0°, 90°, 180°, and 270° with 
respect to the A/B clock. Then phase can be estimated by 

d> = tan ' 1 Ax) ~ D21(> (i) 

A "Ann 

Ideally, tan H^go / D 0 ) would suffice to recover phase, except that subtracting 
pairs of D observations with light phase 180° apart has the benefit of cancel- 
ing residual biases found in the differential signals. In the sequential phase 
measurement, multiple exposures each of duration S results in lower uncer- 
tainty in depth data, whereas, in parallel operation, one exposure captures all 
four phases and the motion blur effects are minimized. 
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In sequential mode, since four exposures are required, it would seem that the 
EPT range camera can output a result only once every 4S seconds. However, 
since the light source modulation is cyclically varied in 90° steps, phase can 
actually be computed using the formula at the end of any one exposure 
because the last four exposures have four different modulation phases. 

In an ideal system with co-located point light source and sensor, with non-dis- 
torting optics of known focal length, and perfect square wave signals, it would 
be possible to convert 0 directly to radial distance r and then to X, Y, Z coor- 
dinates in space by simple geometry. In practice, 0 = 4;r r f / c however, does 
not hold exactly, due to pixel-to-pixel variations in gate timing, imperfect 
square wave signals, and use of a distributed light source. Moreover, a real lens 
distorts the field of view. Accordingly, a per-pixel calibration function may be 
used to map from measured 0 value to Z. This function may be determined 
empirically for each pixel by measuring 0 as a planar test target is moved to 
known Z positions. In addition, a calibration function can be used to map from 
each pixel (row, column) to a 3D direction ray (X/Z,V7Z,1). That function is 
obtained by imaging a grid pattern of known dimensions. Scaling that vector 
by Z yields the desired 3D coordinates. By these means, the EPT range cam- 
era is capable of reporting accurate position information in an absolute sense, 
not just relative, when required by an application. 

Active brightness, a measure of the strength of the received modulated signal, 
can be defined as : 



J A V(A> -Aso ) 2 (A X1 - Avo ) 2 



If active brightness in some pixel falls below a threshold, 0 is deemed unreli- 
able and no coordinates are returned. This feature is important because it 
informs the application on the confidence level of the depth data that is 
returned in the pixel. The application can take corrective actions by recon- 
structing the depth from other good pixels, or take a default action related to 
the application. 



4 Performance Characterization 

In this paper we describe range cameras for two applications that mate the 
160x120 pixel detector chip with appropriate optics and light source. The 
geometric measurement performance of a range camera for occupant sensing 
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has been measured. The uncertainty of the backup obstacle warning camera 
has been predicted using a system model. 

Ongoing work includes characterizing the measurement performance of the 
EPT range camera over the full range of sunlight illumination and temperature 
conditions expected in an automotive environment, as well as verifying the 
frame rate and latency. 



4.1 Occupant Sensing System 

For the front passenger occupant sensing application, a lens was chosen to 
cover 104° vertical x 81° horizontal. Due to field distortion, the relative reso- 
lution is raised in the image center to 1.6 pixels/ 0 . A light source was con- 
structed from forty IR LEDs, radiating 5 W peak optical power into a volume 
matching the lens field of view. 

In order to measure the geometric performance of this EPT camera module, a 
planar test target with IR reflectivity of 0.5 was placed at six different dis- 
tances from 15 cm to 105 cm. For each placement, sequences of 40 frames 
were acquired by the range camera for different values of exposure time S up 
to 32 ms. 

For analytical purposes, forty-three 20 cm x 20 cm regions were chosen with- 
in the occupant detection volume at the six distances. Within each region, the 
metrics defined in 2.1 are calculated for the best S for that region. (The best 
S is selected as the exposure time minimizing the Z uncertainty metric. 
Typically S is 32 ms, except for brightly lit regions that could saturate with 
such a long exposure.) 

Fig. 2 shows histograms of the performance metrics. Z uncertainty varies from 
0.3 cm to 1.5 cm for different analysis regions, due to non-uniformity of the 
light-gathering power of the lens, non-uniform radiation from the active light 
source, and 1/r 2 fall-off of power with distance. The median of each metric in 
each dimension meets the goals stated in 2.2. 

In the sensor chip, the maximum non-saturating signal with 50 common mode 
resets is 2xl0 5 ADC units, while the noise level in the signal is around 5, yield- 
ing a raw dynamic range of 92 dB. On the other hand, for range cameras a 
more relevant metric is the range of signal levels that satisfies a measurement 
performance goal. A regression model can be fit to the relationship between 
active brightness and Z uncertainty, predicting that l A = 480 suffices to achieve 
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uncertainty of 1 cm. On that basis, the dynamic range for satisfying the per- 
formance specification is 52 dB. 
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Fig. 2. Histograms of Geometric Performance Metrics for Occupant 
Sensing EPT Range Camera 



4.2 Backup Obstacle Warning System 

We have developed a system model to predict the depth precision for a given 
active light source, target object, and optics, or alternatively the locus of 3D 
points where a specified precision is achieved. The model could be inverted to 
design the light source to achieve a specified precision on a given locus. The 
parameters of the model include: 

► Light source model: optical wavelength, total power, angular distribu- 
tion (assumes point source co-located with sensor) 
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► Object model: reflectivity and radial distance (assumes surface is per- 
pendicular to line of sight) 

► Optics model: focal length, F/ # , transmission losses of lens and IR filter 
(assume no vignetting or field distortion) 

► Pixel model: pixel area and quantum efficiency 

► Analog to digital converter model: quantization step, voltage swing, 
exposure time 

► Range estimator model: modulation frequency 



FPS:20 Reflectivity. 0.10 Freq:52 LS Power: 10. 00 Watts 




Fig. 3. Iso-Uncertainty Contours, showing loci of 20, 40, 60, 100 mm 
uncertainty. 

The object model converts the emitted light to received light at the lens, which 
the optics model converts to light power falling on a pixel. Assuming received 
and emitted light are in phase, the pixel model converts light power to a volt- 
age, which the ADC model converts to a digital differential signal D max . 
Depending on actual phase, D will vary from 0 to D max in steps of 1 . The range 
estimator model converts this to the corresponding step in radial distance. 

Fig. 3 illustrates the result of the model assuming 10 W of light source power, 
52 MHz modulation frequency, 0.1 object reflectivity, 90° field of view and 20 
frames per second. The figure shows iso-uncertainty contours for 20, 40, 60 
and 100 mm values as a function of distance from the sensor and the light 
source. For instance, the lower curve shows the set of points with uncertainty 
of 20 mm. Clearly, the points that are closer to the sensor and aligned with the 
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axis of the camera reflect more light to the sensor and thus produce better 
uncertainty values. The shape of the curves is influenced by the profile of the 
light source and the well-known cos 4 -effect. If necessary the light source can 
be designed to compensate for the cos 4 -effect by increasing the concentration 
of the light on the boundaries of the field-of-view. 

For the backup obstacle warning system, this configuration of the camera 
shows that distances for objects as far as about 3 m (see arrow A in Fig. 3) in 
the direction of the center axis of the camera can be measured with 2cm 
uncertainty. At center axis of the camera, 1.5% uncertainty requirement is 
easily met (2.3). In the perimeter of the field-of-view, the model suggests that 
the present light source only gives about 6 cm uncertainty (see arrow B in Fig. 
3) for objects in a plane that is 3 m from the back of the vehicle where the 
camera may be mounted. The model suggests that an alternative light source 
with higher concentration of the light in off-center regions would be better 
suited for this application. We observe the clear advantages of using a system 
model for sensor performance analysis. The model provides the information 
that is needed for the camera system designer to tailor the camera and its light 
source for a vehicle application to achieve cost/performance goals of the appli- 
cation. 



5 Conclusions 

This paper described the advantages of Electronic Perception Technology, a 
high frame rate focal plane array 3D sensor, for automotive safety systems. 
Performance metrics and application-driven requirements were presented for 
two specific systems, occupant sensing and backup obstacle warning. We 
detailed the principles of operation of the range camera. We demonstrated the 
use of actual measurement and the use of analytical modeling to show how the 
two configurations of the camera module satisfy the requirements. 
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Abstract 

Future demands for very low emissions from diesel engines, without 
compromising fuel economy or driveability, require Engine 
Management Systems (EMS) capable of compensating for emissions 
dispersion caused by production tolerances and component ageing. 
This paper describes a collaborative project between Ricardo and 
General Motors (GM), aimed at reducing engine-out emissions dis- 
persion and enabling alternative combustion modes, such as 
Ricardo's Highly Premixed Cool Combustion (HPCC), in real-world 
scenarios. Cylinder pressure based control is a key technology 
enabler for achieving the necessary high-level coordination of fuel, 
air and EGR in order to meet the conflicting performance require- 
ments of current and future diesel engines. The paper describes the 
background to the project, the development toolchain using produc- 
tion intent sensors, control strategies developed for both testbed and 
vehicle application and preliminary results obtained. 



1 Introduction 

Future demands of very low emissions on diesel engines are presenting a sig- 
nificant challenge to the automotive industry. The need to remain cost com- 
petitive has deferred the adoption of expensive Nitrous Oxides (NO x ) 
aftertreatment systems thereby placing pressure on achieving even lower 
engine-out emissions. This has led to greater influence of manufacturing tol- 
erances and aging on emission spread. Fig. 1 illustrates the effect of produc- 
tion tolerances on the NO x /Particulate matter (PM) emissions dispersion cloud. 
Although the levels of emissions required for EURO V compliance are achiev- 
able, when current production tolerances are considered the resulting disper- 
sion clouds are significant. 

The Advanced Diesel Engine Control (ADEC), a collaboration between Ricardo 
and GM, is aimed at investigating the feasibility of applying advanced control 
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strategies to reduce the emissions dispersion incurred by production toler- 
ances and ageing whilst maintaining competitive fuel consumption. This 
began with a sensor feasibility study which included a number of new sensing 
technologies appropriate for future mass production. Two sensor types were 
selected to demonstrate varying degrees of benefits versus sensor technology 
cost. These were based on two sets of engineering targets for engine-out emis- 
sions of 0.15 g/km NO x/ 0.03 g/km PM and 0.08 g/km NO x , 0.03 g/km PM over 
the NEDC cycle. The engine technology applied to achieve these levels, name- 
ly advanced EGR cooler with bypass, high specification turbocharging and 
model based air-path control, had already been demonstrated without the use 
of combustion feedback control in a prior Ricardo project [1]. 




NOx 

Fig. 1 . Effect of production tolerances on the NO x /PM emissions dispersion 
cloud relative to EURO V guideline emissions limits 

The objectives of the ADEC Project are: 

► to investigate sources of emissions dispersion and sensitivity of engine 
emissions to key input variables and internal states 

► to investigate sensor technology to support advanced control strategies 
for combustion and charge feedback 

► to compare available sensors and select options for demonstrating the 
above emissions targets 

► to develop new control architectures and functions for the sensor 
options 

► to validate the potential of the two selected systems for addressing 
emissions dispersion on a testbed and vehicle 
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2 Engine Application 

The engine application is a 1.9 litre Family B engine. A schematic diagram of 
the base engine is given in Fig. 2. 




Fig. 2. Schematic diagram of 1.9 litre Family B engine, the base engine for 
the ADEC programme 



3 Sensor Feasibility Study 

A sensor feasibility study was carried out at part of the project. The objectives 
of this task were: 

► to investigate sensor technologies required to support advanced control 
strategies for combustion feedback control 

► to compare available sensors and select options for demonstrating feed- 
back control in order to meet emissions targets 

A total of nine different sensors covering each of the following technologies 
were tested to evaluate their potential for use in closed-loop control of Diesel 
combustion: 

► Direct in-cylinder pressure sensing 

- Piezo-electric 

- Piezo-resistive 

- Optical measurement of deflected membrane 

► In-direct cylinder pressure sensing 

- Deflection of cylinder head 

- Change in cylinder head sealing gap (Head Gasket) 

► Non-cylinder pressure sensing technologies 
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- Ion current (combustion) sensing 

- Block mounted accelerometer 

Direct measurement of cylinder pressure would certainly provide the best 
means of estimating combustion states such as P max , Indicated Mean Effective 
Pressure (IMEP), angle of 50% fuel mass fraction burn (B50). However, in view 
of the cramped conditions inside the 16-valve diesel engine, it was clear that 
any cylinder pressure sensor required for mass production would have to be 
integrated within an existing engine component. The direct sensors were 
therefore chosen to be glow-plug mounted types. Although it is true that the 
term "direct" becomes less meaningful if the sensor is mounted further away 
from the cylinder in such a way as to measure the deflection of the glow-plug 
probe (compared to the traditional flush-mounted Kistler sensor), for the pur- 
pose of this study, in order to differentiate them from sensors used to measure 
cylinder head deflection, they were considered as "direct sensors". 

The response characteristics of the cylinder pressure candidate sensors were 
compared with a Kistler 6053BB60 reference sensor. Figs. 3 and 4 show asso- 
ciated Lissajous for five such glow-plug pressure sensors, depicting the spread 
of candidate versus reference sensors at extreme operating conditions from 
low to high speed and load. These results provided useful information concern- 
ing signal repeatability, linearity, hysteresis, etc. 




0 10 3P » 





Ertr?rn;» H*Jllf 5*»W 



/ 




7 




Ml 


JF 

/ 


^ ' uwere 


10 

a 


/ 

/ QWBN 



D W 30 JU U D III 30 T.I 4J 

R^rtruf ftTlbr SF'VK Hrtr-fm-f Wsflp 5fn*?r 




D II JJ *J 




' i:i JO JO 4.1 



Fig. 3. Lissajous plots (candidate sensor output vs. reference sensor out- 
put) for five different candidate cylinder pressure sensors at 
1 500rpm, 2bar BMEP 



Ricardo's Cylinder Pressure Processing Tool (CPPT), which incorporates the 
Heat Release algorithm, HREL, was then applied to estimate the various com- 
bustion states. This was performed for each of the five candidate sensors using 
both filtered and un-filtered signals at six operating conditions across the 



Coordinated Cylinder Pressure Based Control for Reducing Diesel Emissions Dispersion I 227 



speed/load range. Results were compared to the reference sensors and a 
regression technique was applied to determine the correlation. 
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Fig. 4. Lissajous plots (candidate sensor output vs. reference sensor out- 
put) for five different candidate cylinder pressure sensors at 
4000rpm, Full Load 



The correlation data was combined with other factors including product devel- 
opment maturity, production cost and ease of integration into vehicle electron- 
ics/ECU and signal processing/filtering requirements to select the target sen- 
sor option for the further phases of the programme. 

Cylinder pressure traces from in-direct cylinder pressure sensors suffered 
from noise and interference from combustion and valve events in neighbour- 
ing cylinders, making engine state estimation more problematic and reducing 
the correlation with the states estimated from the reference sensors. A pres- 
sure sensor integrated in a glow-plug (utilising direct measurement of the in- 
cylinder pressure) was therefore selected as the target sensor option. These 
sensors were carefully designed to overcome a number of technical issues 
including robustness to temperature variations related to glowing/non-glowing 
action as well as thermal shock, and minimal noise interference. 



From non-cylinder pressure type sensors, it is only possible to estimate tempo- 
ral (combustion phasing) engine states such as Start of Combustion (SOC). 
Taking this into account, and combined with their low production cost, block 
mounted accelerometers were selected as a low-cost alternative option for 
evaluation. Details of this study will be published at a later date. 
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4 Control Strategy Development 



Cylinder pressure feedback control has attracted much attention in recent 
years, as there are many potential benefits [2, 3]. This section describes the 
development of such a system as the target sensor option to meet the pro- 
gramme objectives. 

The control strategy was defined based on a number of system requirements. 
These include: 

► High-level coordination of fuel and air in order to meet the conflicting 
demands of fuel economy low emissions and good driveability 

► Adaptability to production tolerances and aging 

► Modularity to allow transferability to different sensor types 



A top-level system diagram of the control strategy is given in Fig. 5. 



Fuel Quantities 
Fuel Timings 




Fig. 5. Control Strategy Overview 



4.1 Estimation of Combustion States 

Certain key combustion states were estimated for feedback control. Unlike the 
offline estimation strategies applied in CPPT as indicated in section 3, this 
necessitated approximations appropriate for real-time computations. 
Furthermore, these were applied to each successive engine cycle thus requir- 
ing moving average filtering of the states. Any consistent errors based on the 
approximations applied could be calibrated out. 
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Combustion state estimation algorithms were also developed around the sens- 
ing technology applied. The objective was to avoid the imperfections in the 
sensor signal or to compensate for them. This allowed the use of less refined 
sensors more appropriate to near-future mass production. 

The combustion states considered were: 

► Crank angle at which 50% of the total fuel is burned (angle of B50) 

► Peak cylinder pressure (P max ) 

► Rate of Combustion (ROC) 

► Combustion Instability (CIS) 

► Indicated Mean Effective Pressure (IMEP) 

► Top Dead Centre (TDC) 

The angle of B50 is estimated based on the Rassweiler & Withrow approxima- 
tion [4]. CIS is any appropriate measure of cycle-to-cycle variation of some com- 
bustion state related to the given cylinder. TDC is estimated during overrun to 
compensate for crank-sensor offset. 



4.2 Fuelling Control 

Fuel quantity and timing are controlled separately for each cylinder. Start of 
Injection (SOI) is controlled to match a target angle of B50 for good fuel effi- 
ciency and, together with mass airflow/EGR, low NOx. Fig. 6 describes the 
feedback controller that compensates for both injector and combustion uncer- 
tainties. 




Angle of B50 
Estimation 



Sensor 

Uncertainly 



Fig. 6. Angle of B50 control for each cylinder 
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Although sensor uncertainty is also introduced, this is mitigated by the 
Rassweiler & Withrow approach applied which does not require sensor gain 
calibration. 

During normal fuelling operation, P max and CIS are maintained within limits as 
described below. 



4.3 Air Path Control 

The air path controller controls EGR and boost pressure. EGR is controlled as 
standard by adjusting the EGR valve and throttle to match a target Mass 
Airflow (MAF). Corrections are also made to compensate for variations in 
intake manifold conditions and to maintain combustion stability (see below). 
The EGR cooler bypass valve is actuated using a model-based estimate of the 
intake manifold temperature in order to maintain the correct EGR tempera- 
ture. The boost pressure is closed-loop controlled at high loads using the VGT. 

4.4 ADEC Coordinator 

At the heart of the control strategies is the ADEC Coordinator. This high-level 
coordinator adapts control setpoints harmoniously in order to meet the various 
conflicting performance requirements as well as responding rapidly to any 
sensor failure (Fig. 7). 



Adaption of Fuelling Setpoints 



w 



o 

O Sensor Failure 




Engine Speed, Load, 
Operating Mode, etc. 



Sensor Failure Flags 



Pmax 




Adaption of Air Path Setpoints 



Fig. 7. ADEC Coordinator 
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To ensure stable operation at high levels of EGR, and when running in alterna- 
tive combustion modes (e.g. HPCC) both fuelling and air path controllers are 
adapted. Once combustion instability increases above a given threshold, the 
target B50 angle is immediately advanced for rapid stability control followed 
by a slower increase in target MAF as illustrated in Fig. 8. 

In order to protect the engine, the P max of each cylinder is controlled separate- 
ly within a limit by altering the fuel commands. Firstly an offset is applied to 
the target angle of B50 to directly retard the combustion once the P max limit 
has been exceeded. If the retard authority limit (due to exhaust temperature 
limitation) is reached and further reduction is required, the fuel quantity is 
reduced accordingly as illustrated in Fig. 9. Control action is therefore applied 
in two stages: combustion retard, then fuelling reduction, in order to minimize 
the impact on torque felt by the driver. 



MAF 

increase 




Fig. 8. Combustion Stability Control 




Fig. 9. P, 



Control 
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There is no conflict in timing between P max and CIS control as the CIS is active 
only at part-load. 

An ROC controller is also integrated within the ADEC Coordinator. 



5 Control Strategy Implementation and Testing 

5.1 Prototyping System 

One of the key enabling factors for the combustion feedback strategies based 
EMS is a suitable hardware prototyping system. To obtain the appropriate res- 
olution of the combustion state estimates a high quantity of data per engine 
cycle had to be processed in real time - much higher than today's ECU tech- 
nology. 

To this requirement the prototyping EMS was based around Ricardo's high- 
powered ECU platform, rCube [5]. This enables one-degree sample resolution 
over the combustion phase as high resolution was necessary for effective ROC 
estimation. All combustion states are estimated once per cycle for all four 
cylinders in the rCube as shown in Fig. 10. rCube utilises two processors to 
split between fast interaction with external world (MPC565) and computation- 
ally extensive tasks of combustion states estimation performed by MPC8260. 



i m 1 1 1 1 1 1 1 1 1 




J. 



ESIOGO 
ADEC Strat, 



Fig. 10. rCube Dual Processor System Architecture 



Calculated combustion states are fed over CAN to an ETAS ESI 000 which pro- 
vides the necessary closed-loop control, interfacing with the Bosch EDC16 over 
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ETK and overriding the production fuelling and air-path control commands. 
The schematic layout of the control system is presented in Fig. 11. 




Fig. 1 1 . Rapid Prototyping System Architecture 



5.2 Control Strategy Implementation and Testing 

The control strategies went through several stages of development. Firstly 
simple concept studies were carried out in MATLAB® based on pre-recorded 
cylinder pressure data in order to identify suitable estimation techniques suit- 
able for real-time application. The sensitivity of combustion states to signal 
amplitude resolution and crank sample resolution were also investigated. 

The second stage involved Software-in-the-Loop (SiL) testing. Some of the com- 
bustion state estimation algorithms were implemented together with feedback 
control in SIMULINK® and tested using a rapid in-cycle engine model devel- 
oped by Ricardo. This simulation tool modelled the pressures waves along the 
entire engine air path thus enabling realistic cylinder pressure signals neces- 
sary for feedback control development to be simulated. This model also incor- 
porates multi-injection capability and was tailored for the specific application 
to include random combustion variations dependent on operating conditions. 
Fig. 12 indicates a top-level SIMULINK diagram of the SiL environment. 
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Production ECU 




Simulated Combustion States for All Cylinders 

Fig. 12. Software-in-the-Loop Testing Environment 

In the final stage, the control strategies were implemented in ASCET-SD® to 
run on the Prototyping System. This was first tested in a dSPACE Hardware- 
in-the-Loop environment before commissioning on the testbed. 



6 Test Results 

Preliminary results have been obtained to demonstrate the performance of the 
various control algorithms. Specifically these were angle of B50 balancing, 
Pmax an d stability control. These were tested first using reference Kistler sen- 
sors (Kistler 6053BB60) followed by the glow-plug integrated pressure sensors 
selected in the Sensor Feasibility Study. For the purpose of this paper the glow- 
plug sensor results are presented for B50 balancing, P max control as well as for 
the combustion stability control. 



6.1 Angle of B50 Control Results 

The angle of B50 controller performance is demonstrated in Figs. 13 and 14. 
The first of these indicates the effect of switching the controller on and off 
under steady-state conditions. It is clear how the spread in B50 is reduced 
accompanied by a corresponding spread in SOI as the fuel timings are individ- 
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ually adjusted in order to balance the combustion phasing. The second figure 
demonstrates how the controller is able to track changes in the B50 set-point. 




Fig. 13. Steady-state angle of B50 control 




Fig. 14. Transient angle of B50 control at 2000 rpm, 76 Nm at 1500 rpm, 
30 Nm 



6.2 P max Control 

The control of P max in response to an increase in driver demand is demonstrat- 
ed in Fig. 15. Two tip-in/tip-out manoeuvres were applied, first without then 
with P max control. The results clearly demonstrate how P max is successfully lim- 
ited by a combination of fuel retard and fuel quantity. Here calibration thresh- 
olds for P max and SOI retard limit were set purely for demonstration purposes. 
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Fig. 15. P max control results at 2000 rpm, accelerator pedal 60-100% 



6.3 Combustion Stability Control 

Maintaining stable combustion in response to forced instabilities was demon- 
strated successfully. At high EGR levels, the timing was suddenly retarded on 
all cylinders resulting in an increase of CIS above a given threshold. As indi- 
cated in Fig. 16, the target B50 angle was immediately advanced to compen- 
sate followed by an increase in target MAF triggered by saturation of the B50 
advance controller. 
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Fig. 16. Combustion stability control at 1500 rpm, 2bar BMEP 



These are gradually wound down to their nominal set-points until the next 
threshold crossing occurs (compressed timescale for demonstration). 
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7 Conclusions 

This paper describes the on-going developments in the ADEC Project, a collab- 
oration between Ricardo and General Motors. The conclusions so far are as fol- 
lows: 

► a sensor feasibility study has identified a glow plug mounted pressure 
sensor type as the target sensor option 

► a cylinder pressure feedback control system was developed and has 
demonstrated successful control of angle of B50 timing, P max limiting 
for engine protection and good control of combustion stability 

The next steps in the project are as follows: 

► Validation of robustness of control system to dispersion tests carried out 
on the testbed 

► Demonstration of control system on a demonstrator vehicle 

► Application of accelerometers sensor strategies 
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Abstract 

This paper presents a High Temperature Floating Gate MOSFET 
Driver ASIC for on-the-engine Injector Control. The circuit was 
specifically designed for this harsh environment and has been real- 
ized using Silicon-on-Insulator (SOI) Technology. The first section 
describes the operation of the system sub-blocks: low- and high-side 
drivers, the current sense and the fault detector. In the second sec- 
tion, measurements show the proper operation of the circuit up to 
225°C. 



1 Introduction 

Placing the injector control electronics on the engine allows removing cables 
thus reducing weight. Moreover, having the control electronics on the motor 
could enable the stand-alone testing of the engine. The challenge is to have 
electronics that can reliably sustain the engine harsh environment, especially 
the large temperature range (-40 °C to +200°C). Silicon-on-Insulator (SOI), pro- 
viding fully dielectric isolation and leakage reduction, has already demon- 
strated to be the best technology in designing high temperature analogue and 
digital circuits [1, 2]. This paper presents a prototype of low side and high side 
power MOS drivers integrated in a system-on-chip ASIC. It has been realized 
in a high temperature and high voltage SOI process with tungsten metalliza- 
tion for long-term reliable operation at high temperature. Besides drivers, cur- 
rent sense and fault detector circuits have also been implemented in order to 
have all analogue functionalities required for driving power MOS on inductive 
loads at high temperature. 



2 System Overview 

Fig. 1 depicts the ASIC prototype schematic and layout. Beside the digital con- 
trol part that will not be considered hereafter, the ASIC includes three main 
analogue functionalities: the drivers, the current sense and the fault detector. 
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2.1 Drivers 

Drivers are required in order to interface the digital control with external 
power MOSFETs. Two kinds of drivers can be distinguished in Fig. 1. The first 
and simplest one, called the low-side (LS) driver has its output voltage referred 
to ground. In the ASIC prototype, this LS driver is controlled by digital 5 V 
logic. Its output can swing between 0 and 10 V levels versus ground. The sec- 
ond driver, called the high-side (HS) driver is more complex. Its input is also 
controlled by digital 5 V logic and its output can also swing 10 V. The differ- 
ence is that the output is completely floating versus the ground. This allows 
keeping the POWER MOSFET turned ON with its source connected to an induc- 
tive load. An external capacitor is required for the HS driver in order to be able 
to drive the gate of HS MOSFET with a voltage higher than the supply voltage. 
This way, the ON resistance of the HS power MOSFET is always kept very low. 
For applications where the HS MOSFET has to be turned ON for a long time 
(i.e. without switching it), an internal charge pump has been integrated in the 
driver in order to maintain the charge in the external capacitor. Measurement 
results are shown in section 2. 





Fig. 1 . ASIC prototype (digital control + drivers) schematic and layout 



2.2 Current Sense 

In order to control the LS and HS drivers, the digital controller needs to know 
the actual current flowing across the inductive load. The Current Sense (CS) 
circuit is responsible for comparing this current (measured across a grounded 
sense resistor in Fig. 1) with a digital threshold. The comparison result is a fed 
back to the digital controller. Therefore, the current sense block is implement- 
ed by a "mixed-mode" comparator, comparing an analogue voltage with a dig- 
ital threshold D, which is a much simpler way than those used so far. In the 
implemented prototype, D is an 8-bit word. Fig. 2 depicts our CS measurement 
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at 25G and 200G for 8 bits digital threshold arbitrarily fixed at D12 8 and 

D25 5 respectively. The comparator needs no more than 1.4 pto switch at 
200C. The measured thresholds for D1=2 8 and D255 are also very close to 
the theoretical values. Note that the CS 5 V reference voltage is internally gen- 
erated with a buffered 5 V voltage reference. The supply voltage must be at 
least 6 V in order to correctly generate the internal 5 V reference voltage. 



T=025°C T=200°C 




Fig. 2. Current sense measurements at 25C and 200C for digital thresh- 
old of respectively Dl=28 an D255 



2.3 Fault Detector 

In order to avoid inductive load or battery damaging, it is important to verify 
that the power MOSFET's operate as expected. A simple way to check this is 
to monitor each power MOSFET drain-to-source voltage ( V ds ). When the digital 
controller requires turning ON a MOSFET, it is expected that V ds of this device 
drops under a threshold voltage after a small delay. Similarly, when the MOS- 
FET is turned OFF, the y ds voltage should be higher than this threshold. Any 
mismatch between the expected MOSFET state (ON or OFF) with the actual 
measured V ds is interpreted as a failure. In such case, the digital controller 
should turn OFF all the MOSFET's in order to avoid any damage caused by an 
over-current. The Fault Detector (FD) is also simply implemented by a "mixed- 
mode" comparator, comparing a floating differential voltage ( V ds ) with a pre- 
defined digital threshold. In this prototype, the digital threshold can be set to 
2 V or 2.4 V. Fig. 3 depicts quasi-static measurements at 25C of the compara- 
tor output voltage versus the differential input voltage ( V ds ). The common 
mode (CM) of this differential voltage is swept from -1 up to 50 V. The meas- 
ured threshold is very close to the expected 2 V. We only observe a small influ- 
ence of the CM voltage on this threshold. (50 V shift on CM corresponds to 100 
mV shift on threshold, i.e. A threshold /A CM 2= mV/V). The FD switching time is 
shorter than 1 p over temperature. 
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Fig. 3. Measured fault detector output voltage versus differential input 
voltage as function of the input common mode voltage at 25C. 




(2.2Q) 



Power MOSFETs 
(IRF540N 



Driver 

^-injector ("1 OmH ) 



Fault 

Detector 



Simplified 

Logic 

control 



Fig. 4. Full system measurement setup (Driver, current sense, fault detec- 
tor). Only the driver can be under thermal stream on this board. 



3 System Measurement 

The full system, including a HS driver, a LS driver, a current sense and a fault 
detector has been tested (ASIC of figure 1 has been sub-diced in order to test 
separately the analogue parts). In the setup, only the driver can be heated up 
to 225C with a thermal stream. Other parts are at 25C. However, they have 
been previously tested separately up to 22 5C. External components of Fig. 1 
setup are: 

► Power MOSFETs: IRF540 

► Inductive load: 10 mH 

► Sense resistor: 2.2 Q 
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Fig. 4 shows the test measurement board. Digital control is such that the cur- 
rent target is set to 100 mA during 300 pi and to 60 mA during 600 p before 
turning off. Note that the current levels in the inductive load and the sense 
resistor represent a scaling down with respect to the final application (few 
Amps). This scaling down is acceptable since the driver load is a MOSFET able 
to drive up to 33 A. 

Fig. 5 is a screen shoot of the sense resistor voltage, the HS MOSFET gate-to- 
source voltage (Vy and the HS fault detector output. Measurement results are 
very similar at 256 and 2256. Fig. 5 is relative to a 12 V battery voltage. 
However, the driver can sustain battery voltages up to 55 V at 22 56. 



Sense 

Resistor 

Voltage 



HS V GS 



HS Fault 
Detector 
Output 

Fig. 5. Measured sense resistor voltage, HS V gs and HS fault detector out- 
put (from top to bottom) at 2256 with a 12 V battery voltage 
(Results are very similar to those at 256). 

From Fig. 5, we can see that the fault detector output (bottom wave) is in 
accordance with the MOSFET gate-to-source voltage (middle wave). The sys- 
tem is working correctly. 

Concerning delays. Fig. 6 shows a zoom on one HS switching at 2256. Waves 
are the sensed voltage, the voltage on HS MOSFET source and the HS V gs from 
top to bottom respectively. Switching time is between 1 p and 2 p with the 
used MOSFET (IRF540). Results are similar for the LS driver. 
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Fig. 6. Zoom on one HS switching period at 225C with 12V battery volt- 
age. Waves are the sense resistor voltage, the HS MOSFET source 
voltage and the HS V gs (top to bottom wave respectively). 



4 Conclusion 

Taking advantage of the full dielectric isolation and the high temperature per- 
formance of SOI Technology, all the blocks of Floating Gate MOSFET Driver 
have been successfully designed, fabricated and tested up to 2250. This ASIC 
shows the opportunity to place the Injector Control Electronics directly on the 
engine. This would enable the stand-alone testing of the engine while remov- 
ing cables thus reducing weight. 
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Abstract 

The development of distributed pressure sensors based on conduct- 
ing polymers for automotive application is described. These pressure 
sensors can be placed on large surfaces due to its flexibility and low 
cost. As a consequence, pressure data are obtained not only from 
some discrete points but also from continuous surface points. 256 of 
these sensors have been placed on an automotive seat, giving the so- 
called "smart automotive seat". These sensors are able to provide 
information about the person on the seat (adult, child, size, weight, 
etc.). This information can be used by the central electronic system 
in the vehicle for activation and modulation (or deactivate if neces- 
sary) of security systems (airbag, seat belt's prestress system, etc.). 
Response time is lower than 2 milliseconds, and it works in pressure 
ranges of 0.05-10 kg/cm 2 , showing a lifetime of more than 10 mil- 
lions of actuations. 



1 Introduction 

The field of automotive industry increasingly demands higher accuracy secu- 
rity requirements. As a result, the accuracy of safety measurements in a car 
are progressing enormously. One of the most demanded and needed security 
systems is based on the presence detection, providing information about the 
person on the seat. In this sense, the elucidation of whether the person on the 
seat is an adult, or a child, and the size and weight of this person should led 
the activation or deactivation of security systems (airbag, seat belt's prestress 
system), and in case of activation, the adecuation of its actuation to the specific 
person. 

Until now, no pressure sensor with appropriate performance for its application 
in automotive field has been developed. Traditional presence detecting sensors 
based on metallic wires, permanent magnets or electromagnetic induced cur- 
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rents are adequate to obtain information from discrete surfaces or areas but 
not to inform about a whole surface [1]. Moreover, they are not quantitative 
sensors, but only qualitative (on-off) sensors informing about presence or not 
presence of an object. Recently, a new generation of pressure sensors begins 
to be developed and used for these kinds of applications. It consists on the use 
of piezoresistive polymer film elements, between plastic substrates, showing a 
decrease in electrical resistance when a normal force is applied [2]. These sen- 
sors constitute a new generation of pressure sensors enabling monitoring 
along a whole specific surface area, and not only discrete points pressure mon- 
itoring. However, their use for large surface applications seems unlikely due to 
the high cost of materials used. The electroactive materials incorporated in 
currently developed configurations use metallic microparticles (silver) embed- 
ded on a polymer film or metallic central films (silver) separated by a pressur- 
able elastomeric polymer film. In this sense, some devices which are now in 
commercial state have been already developed, showing several variations in 
its configurations, being either capacitive [3-4] or resistive [5-6] pressure sen- 
sors. Among resistive ones, Tekscan [5] technology uses a silver ink layer on 
each of plastic electrodes, followed by a pressure sensitive material layer. The 
sensor is formed laminating together two layers of substrate. This configura- 
tion gives a resistivity decrease when different pressures are applied on it. On 
the other side, Peratech [6] technology is based on metal-filled polymers as 
electroactive materials. This composite material behaves like a perfect insula- 
tor in absence of force, and metal like when deformed by compression, twist- 
ing or stretching. As a consequence, an exponential resistance drop with the 
pressure is observed. In spite of these valuable advances, the use of such high 
cost materials as silver is without any shadow of doubt a very important incon- 
venient since large surface applications can not be afforded by these types of 
sensors. 

In this work, the construction of an intelligent seat is described, using a 
patented new technology of distributed pressure sensors [7]. This new tech- 
nology uses conducting polymers as electroactive materials for construction of 
pressure sensors. The use of two thin conducting polymer films onto plastic 
substrates and the presence of a roughness at microscopic level lead a resis- 
tivity decrease when different pressures are applied. This new technology 
shows great advantages for application on high-surfaces, as on automotive 
"intelligent seats": A) Plastic sheets being used final sensor devices are light. 
B) Sensors being low cost, it is thus a competitive technology to be incorpo- 
rated in automotive industry. C) Flexibility in final sensor devices makes feasi- 
ble its incorporation in a flexible surface as it is the case in automotion seats. 

The new sensors are based on ICPs such as polypyrrole (PPy), polyaniline 
(PAni) or polyethylenedioxythiophene (PEDOT) are emerging electroactive 
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materials useful for several industrial applications [8]. ICPs are already being 
employed in electronic capacitors [9], electrochemical (bio)sensor arrays [10], 
polymer light emitting displays [11], through-hole metallization of multilayer 
printed circuit boards (PCBs) [12], long-term solderable lead-free tin surface 
finishes for PCBs [13], transparent antistatic coatings for insulating substrates 
[14], conducting fibres, textiles [15] and anticorrosion coatings [16]. New 
emerging applications being studied are electromagnetic shielding applica- 
tions in electronic equipments [17], conducting hot melt adhesives [18], elec- 
tromembranes [19], cathode materials for lithium-ion batteries [20], elec- 
trochromic devices [21], organic solar cells [22], electroactuators [23] and nano- 
electronic devices [24]. For some of these applications, and specifically for 
application as thin film for pressure sensors, a good processability in ICP is cru- 
cial, in order to get an easy applied polymer film on the substrate. Thus, being 
conducting polymers insoluble in common solvents, aqueous dispersions of 
these polymers are used in these cases. 

Conducting polymer dispersions are usually prepared by chemical oxidative 
polymerization of monomer in water in the presence of a polymer steric stabi- 
lizer [25]. By using different water soluble polymers during the polymerization 
such as poly( vinyl alcohol), poly(ethylene oxide), poly(N-vinylpyrrolidone), cel- 
lulose derivatives and poly(styrene sulphonate) salts, macroscopic precipita- 
tion of the ICP is prevented and sub-micrometer dispersed particles are 
obtained [26]. Reaction conditions such as temperature, oxidant, monomer 
concentration, etc. as well as the chemical structure, molecular weight and 
concentration of the polymer steric stabilizer strongly affect the size and sta- 
bility of the colloidal dispersions and the conductivity of the resulting PPy film 
upon water removal. The presence of the insulating polymer steric stabilizer 
in the dried PPy film leads to lower conductivity values (< 10 1 s/cm) when 
compared to raw PPy. If such PPy aqueous dispersions are employed as addi- 
tives in water borne paints, the conductivity of the resulting coating after dry- 
ing is usually very low (< lO 5 s/cm). In a previous paper [27] new conducting 
polymer dispersions with average particle size below 100 nm are obtained by 
using sonochemical techniques during the oxidative polymerization process. 
These dispersions can be directly applied on substrates obtaining a conducting 
continuous film. 

The development of a new low cost and flexible pressure sensor technology 
based on the deposition of conducting inks onto plastic flexible sheets has pre- 
viously been described [7]. In this work, this conducting polymers based thin 
film technology is used for the construction of a smart automotive seat. Placing 
them on the surface of the seat, pressure data are obtained not only from some 
discrete points but also from continuous surface points. This smart automotive 
seat, based on 256 of these sensors, is able to provide information about the 
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person on the seat (adult, child, size, weight, etc.). This information can be 
used by the central electronic system in the vehicle for activation and modu- 
lation (or deactivate if necessary) of security systems (airbag, seat belt's pre- 
stress system, etc.). 



2 Sensor Construction: Experimental 

Experimental procedures for distributed pressure sensor construction are 
described below: 



2.1 Conducting Ink Preparation 

A conducting polymer dispersion was used as conducting ink for its applica- 
tion on a flexible plastic sheet. Poly(3,4-ethylenedioxy thiophene) (PEDOT) was 
choosen as conducting raw material, and aqueous dispersions of this polymer 
were prepared for the preparation of conducting inks. PEDOT aqueous disper- 
sions were prepared by using an Ultrasonic Processor (model UP 400 S from 
Dr. Hielscher GmbH) during the synthesis. As an example, ethylenedioxythio- 
phene (1 ml, 9.4 mmol) and 3.5 g of poly(styrene sulfonate, sodium salt) were 
dissolved in 100 ml of distilled water. To this mixture, an equimolar amount of 
ammonium peroxydisulphate (6.58 g, 28.8 mmol) dissolved in 50 ml of water 
was added dropwise over a period of 4 minutes. After 1 hour of reaction under 
ultrasonic irradiation a dark blue PEDOT aqueous dispersion was obtained. 
Dispersion particle sizes were measured using a Beckman Coulter N5 
Submicron Particle Size Analyzer. 



2.2 Conducting Flexible Sheet’s Preparation 

Conducting flexible sheets were prepared placing a PET (polyethylenetereph- 
talate, 5x5 cm 2 in surface, 125 pm in thickness) plastic sheet on a spin coater. 
50 pi of conducting polymer ink were dispensed on it and it was rotated at 
1 500 rpm. After 1 minute, it was stopped and left drying at room temperature. 
A thin conducting film was created onto the plastic sheet, giving a conducting 
flexible plastic sheet. 
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2.3 Sensor Configuration 

In the construction configuration adopted, an additional flexible plastic sheet 
is also used together with the conducting polymer sheet, with conducting and 
non-conducting paths printed on it. The conducting paths act as electrodes, 
enabling the electrical connection of the different sensor elements, and they 
are printed on the sheets by either serigraphy or photolithography. The con- 
figuration and distribution of the conducting paths determines the size of each 
sensor element and their spatial distribution. The insulating paths allow the 
working pressure range of the sensor to be controlled. The configuration used 
for incorporating the sensors to the seat was a 4 x 4 array of sensor elements 
with an element size of 1 cm 2 . 



2.4 Interface Software 

Real-time graphic representation of the information was acquired creating an 
application in the Labview programming environment. The current values 
measured by each of the sensor elements were converted to pressure data, 
using the calibration values previously obtained by the sensor configuration to 
make this conversion. The value of each of the pressure elements is displayed 
in real time on a colour scale going from blue (minimum pressure) to red (max- 
imum pressure). The program thus enables real-time display of the pressure 
being exerted at each of the different points of the sensor. 



3 Results and Discussion 

Aqueous conducting polymer dispersions synthesized with mechanical stir- 
ring led to a broad distribution of particle size from tens of nanometres to sev- 
eral microns. When applied by sping coating onto plastic sheets for fabrication 
of conducting films, surface inhomogeneity of the coatings is significantly 
enhanced as a consequence of the presence of large conducting polymer par- 
ticle sizes. The objective for a pressure sensor construction being an homoge- 
neous and continuous conducting plastic sheet, lower particle sizes have to be 
obtained in starting conducting polymer dispersion. For this purpose, sono- 
chemical techniques such as ultrasonic irradiation have been employed dur- 
ing the synthesis of the conducting polymer dispersion (PEDOT dispersion) in 
the presence of ammonium peroxydisulphate, as oxidant, and poly(styrene sul- 
fonate) as steric stabilizer and dopant, as already described in our previous 
work [28]. Fig. 1 illustrates the particle size distribution obtained using the 
sonochemical technique. In this case, the average size of the particles in the 
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ultrasonic irradiated PEDOT dispersion is about 70 nm as determined by Light 
Scattering (LE). 



10 40 70 100 130 160 190 220 250 290 

Size (nm) 




Fig. 1 . Particle size distribution obtained in a sonochemical synthesis pro- 
cedure 

This PEDOT dispersion has been deposited by sping coating onto plastic flexi- 
ble sheets obtaining a nanosized thin conducting film on it. Fig. 2 shows the 
surface picture obtained by atomic force microscope of these conducting films. 




Fig. 2. Surface picture of conducting plastic film obtained by Atomic Force 
Microscope 

A surface roughness can be observed at microscopic level, with an average 
grain size of about 50 nm, in good agreement with LS measurements. This 
roughness at microscopic level is thought to be the main point defining the 
working mechanism in the sensor. A sensor device is constructed by superpo- 
sition of two conducting sheets by leaving conducting film inside. With no 
pressure on plastic sheets, there is no contact between conducting material or 
the contact is extremely light, due to existing microscopic irregularity on these 
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conducting surfaces. As the pressure on plastic sheets increases, the contact 
between the peaks in the conducting surfaces becomes higher and higher. 
Conducting rough surfaces progressively overlap until a full contact is 
obtained at a specific pressure value. In fact, a calibration plot is obtained by 
deposition of different masses on the sensor surface and recording the corre- 
sponding electrical intensity values. As it is shown in Fig. 3, a linear conduc- 
tivity increase (or resistivity decrease) is observed until a constant value is 
reached. This is associated to the full contact point between conducting sur- 
faces. Consequently, the conducting polymer film used as electroactive mate- 
rial in this technology acts as a bifunctional part: on the one hand, as the con- 
ducting material (as the silver in previous configurations); on the other hand, 
as the part allowing more or less contact between sheets with pressure 
changes (the pressurable elastomeric polymer film in previous configurations). 



Fig. 3. 
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Calibration plot for a flexible pressure sensor. Electrical intensity 
values for different masses on the sensor surface 



The conducting plastic sheet and the litographied plastic sheet having con- 
ducting and non-conducting paths printed on it are superposed. Thus, a 4 x 4 
array of sensor elements with an element size of 1 cm 2 is obtained in an uni- 
tary device (Fig. 4). 




Fig. 4. Sensor layer structure 
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Then, 16 of these unitary device elements were used to obtain a total of 256 
measurement points distributed over the surface of the seat (Fig. 5). For their 
integration into the flexible structure of the seat, the sensors were simply 
glued onto it. Their location was chosen taking into account the optimum seat 
parts for providing information about the person on the seat (adult, child, size, 
weight, etc.). Most of the sensors (160 elements) were therefore located on the 
seat base, with the rest distributed over the back. More specifically, 16 of the 
sensor elements were located on the seat headrest, to provide precise data on 
its position. This information could be of great use for automatic headrest posi- 
tion adjustment or optimum opening of the airbag. 




Fig. 5. Sensor location in the automotive seat 

Each of the sensor elements acts electronically as a variable resistor in accor- 
dance with the applied pressure. The range of this electrical resistance varies 
among several kQ at low pressures and a few ohms when maximum pressure 
is applied (before plastic deformation occurs). Multiplexing electronics was 
used to create a sample of the 256 sensors. To maintain the low cost of the sen- 
sors, simple electronics were used based on a voltage divider configuration or 
current-to-voltage converter. A control circuit sequentially connects each sen- 
sor element to a measurement unit, which makes an analogue/digital conver- 
sion of the pressure value obtained. In this way, a sample of the individual 
pressure exerted on each of the 256 sensors in a time of less than 2 ms is 
obtained. 

Using this configuration, we have deduced the calibration parameters relating 
the voltage obtained at the output of the D/A converter (measurement unit) to 
the pressure exerted on the sensor. The data obtained show that sensor 
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response is exponential. The sensor resistance decreases abruptly when a 
determined initial pressure is applied and tends towards a constant final value 
on applying the maximum pressure. Thus, the sensor is designed to work in 
the intermediate part of its operating range where its behaviour is more linear 
and stable. This range is normally between 2 kQ and 100 Q. 

A real-time representation is obtained converting current values measured by 
each of the sensor elements to pressure data. Calibration values previously 
obtained by the sensor configuration were used to make this conversion. The 
value of each of the pressure elements is displayed in real time on a colour 
scale going from blue (minimum pressure) to red (maximum pressure). The 
graphic representations of each of the sensor elements are shown spatially on 
a picture representing the sensorized vehicle seat. The program thus enables 
real-time display of the pressure being exerted at each of the different points 
of the seat. Fig. 6 shows the graphic representation as seen on the computer 
screen. 




Fig. 6. Graphic representation of pressure values in the smart automotive 
seat 

The way these sensors have been constructed allows the spatial configuration 
of the different sensor elements to be easily modified and tailor-made designs 
to be created. For example, sensor elements can be made with different spa- 
tial resolutions and/or non-uniform distribution to adapt to the application's 
requirements. The range of working pressures can be adjusted by altering the 
space between the conducting paths acting as electrodes, and/or the thickness 
or spacing between the non-conducting paths that act as spacers between the 
conducting layer and the electrodes. 
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An aspect to be simplified in future developments is the connectivity of the 
different sensor elements. There are no particular problems with the present 
configuration, as long as a large number of sensors are not used. However, in 
applications that require sensorization of a large surface area with a high spa- 
tial resolution at the same time, it would be convenient to adopt new ways of 
making the connections between the various sensor elements and the meas- 
urement unit. 



4 Conclusion 

A distributed pressure sensor is developed by using conducting polymer thin 
films on flexible plastic surfaces. Conducting formulations can be produced 
and deposited by dip-coating and printing high scale deposition technologies. 
The low cost and flexibility of final sensor units make them suitable for plac- 
ing and adapting them on large surfaces, obtaining pressure data not only 
from discrete points but also from continuous surface points. Pressure ranges 
of 0.05-10 kg/cm 2 and lifetimes exceeding 10 millions of actuations are 
obtained with this technology based sensors, that shows a response time lower 
than 2 milliseconds and are passive elements versus electromagnetic interfer- 
ences (EMI). 

By placing 256 of these sensors on an automotive seat, a "smart automotive 
seat" has been developed. The smart automotive seat is able to provide infor- 
mation about the person on the seat (adult, child, size, weight, etc.). This infor- 
mation can be used by the central electronic system in the vehicle for activa- 
tion and modulation (or deactivate if necessary) of security systems (airbag, 
seat belt's prestress system, etc.). 
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Abstract 

We will discuss the advantages of contact-less infrared thermome- 
ters as main sensor in automotive climate control systems and 
address the practical considerations one should take into account 
when developing such a system. In order to offer a price competitive 
solution with the more commonly known aspirated thermistor sen- 
sor, Melexis developed an integrated infrared thermometer targeted 
at size reduction, improved ease of installation, multi zone tempera- 
ture control and cost improvement. The features of this new product 
are highlighted. Furthermore we discuss the functional and relia- 
bility requirements that have to be met by automotive infrared sen- 
sors and how these requirements are met in a modern sensor design. 
Finally, we show the results of the development of low cost infrared 
array sensors for next generation air-conditioning systems in combi- 
nation with safety applications. 



1 Introduction 

As many people spend several hours a day in their car or truck, it is important 
to provide a good thermal environment, which gives comfort and optimises 
performance for both the driver and passengers. In recent years, infrared con- 
tact-less sensors have become widely accepted and implemented as the com- 
fort sensor for mobile air-conditioning (MAC) applications [1]. In such a system, 
passive long-wavelength (5 to 15 micrometer) infrared sensors based on ther- 
mopile technology are being used. This paper discusses the different applica- 
tions of these infrared sensors in the automotive environment. 

The application of infrared sensors can be split in two broad categories: com- 
fort and safety applications. 

In the comfort application, the infrared sensor is used to track the comfort feel- 
ing of driver and passengers. The signal output of the sensor is used as the 
input to control the settings of the automatic air-conditioning system. We 
describe the specific demands that MAC applications ask from the infrared 
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thermometer and highlight the technical solutions for these requirements. 
There is a growing penetration of the infrared thermometer in the automotive 
air conditioning market. Melexis started volume production of infra red ther- 
mometers 3 years ago. Current volume is more than half a mullion pieces per 
year for 5 different car models of Japanese and US car manufactures. 
Production volume grows in the next two years above one million pieces per 
year for European, US and Japanese car manufactures. 

Safety applications of infrared sensors are not yet widely commercialized. This 
paper discusses the possible applications in an automotive environment, 
derives the requirements for the infrared sensor, and again discusses the tech- 
nical solutions that have been developed. 



2 Automotive Comfort Sensing Application with Infrared 
Thermometers 

An automatic air-conditioning system needs to measure the comfort of the pas- 
sengers in order to take the appropriate air-conditioning action. 

The thermal comfort feeling of the driver or a passenger in a car is a function 
of six parameters: 

► Cabin air temperature 

► Air blowing speed on exposed skin 

► Radiation: sunlight radiation and thermal radiation level inside car 

► Humidity of the air 

► Activity of the person 

► Clothing worn by the person 

When a person is highly active, he needs a lower air temperature in order to 
feel comfortable. For a car climate control system, the activity level of the 
occupants is known and relatively constant. A person driving a car dissipates 
about 160 W which needs to be carried off to make him feel comfortable. 

The amount of clothing a person wears is of course a personal choice and 
depends on the season. A person wearing thicker clothes requires a lower 
operational temperature in order to feel comfortable. Each person will set the 
desired temperature according to his own preferences. This is recognized in 
both references [2] and [3]. Reference [2] gives comfort temperature zones for 
typical summer and winter clothes worn in an office environment. Reference 
[3] lists different types of clothes and their impact, in °C, on the comfort tem- 
perature. 
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The other 4 parameters mentioned above are taken into account by one single 
infrared sensor. The infrared sensor measures the average temperature of the 
surface within its field of view. Depending on the position of the sensor, the 
surface inside the field of view is made up of parts of the car interior, clothes 
of the persons inside the car, and skin of the persons inside the car. 

All of these surfaces change temperature under influence of air temperature, 
air blowing speed and sunlight radiation. These three parameters are thus 
directly measured by the infrared sensor, using the relation between surface 
temperature and infrared radiation emitted by the surface. 

As mentioned in reference [2], humidity is a special case. When the humidity 
is high, the evaporation of sweat becomes more difficult. As a result, the per- 
sons in the car will get a wet skin and wet clothes, and will feel sticky and 
uncomfortable. As skin and clothes are not cooled by evaporation, their tem- 
perature will rise. This rise in temperature of skin and clothes will be sensed 
by an infrared sensor, which will indicate to the air conditioning controller 
that more cooling is needed. Reference [2] mentions that the air temperature 
comfort zone is moved about 1 °C lower when the relative humidity increases 
from 30% to 60%. 

To take into account humidity in the infrared measurement, it is necessary to 
view the persons inside the car, as their surface temperature will be affected 
by relative humidity of the air in the car through the mechanism of evapora- 
tion of sweat. However, it has also been demonstrated that an extremely effec- 
tive and energy-efficient automatic climate control system can be realized by 
measuring only the average surface temperature of the front windshield and 
the dashboard: see references [4] and [5]. By combining infrared sensing with 
an optimized A/C loop, the authors of [5] were able to save 40% to 70% of the 
energy needed for climate control, while maintaining optimum comfort for the 
car passengers. 

As an infrared sensor can take into account the 4 main parameters influencing 
comfort feeling, i.e. air temperature, air-speed, sunlight radiation and humidi- 
ty, it is appropriate to consider an infrared sensor is rather as a comfort sensor 
than as simply a contact-less thermometer. Compared to the more commonly 
used aspirated thermistor, the infrared sensor offers the opportunity to 
achieve an improved, stable thermal comfort for the car occupants. Even 
though the sensor does not measure directly the direction of the sunlight, it is 
possible to take the effect of the sunlight into account as sunlight will imme- 
diately increase the surface temperature. 
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Compared to the often used aspirated thermistor, the infrared comfort sensor 
offers the following advantages to the MAC system designer: 

► The sensor takes into account more parameters which influence the 
comfort feeling of the passengers; 

► The controller receives a measurement of the temperature in the car 
interior, not just of the temperature in front of the dashboard; 

► By nature, the infrared sensor has a small thermal mass, giving a fast 
response to changing temperatures and external factors (thermal time 
constant in the order of 30 ms, digitisation and linearization time 
around 300 ms). This enables the designer to use simples control algo- 
rithms while still achieving a tighter control and stability of the pas- 
sengers comfort; 

► By taking into account more parameters which influence directly the 
comfort of the passenger, a MAC system with infrared sensors can be 
more easily adapted to car model changes as it is easier to adjust the 
MAC control algorithm to changes in cabin design such as window size 
changes; 

► Reduced size and easier integration in the air conditioning control unit; 

► Direct multi zone control using a single sensor with multiple ther- 
mopiles for a marginal additional cost. 



3 Automotive Comfort Sensing Requirements on Infrared Sensors 

In order to allow application in automotive comfort sensing, an infrared sensor 
needs to meet the requirements in Tab. 1 . 

4 Automotive Comfort Sensing Solutions 

4.1 Present Day: Infrared Sensor with Signal Processing ASIC on PCB 

In its development of infrared sensor modules for automotive climate control, 
Melexis was able to meet all of the requirements of the preceding paragraph, 
by making design trade-offs and using advanced mixed analog-digital signal 
conditioning techniques. 

To meet the requirements of reliability, stability over time and wide tempera- 
ture operating range, we decided to use only CMOS materials and processes in 
the manufacture of its thermopile sensors. The design trade-off made concerns 
the sensitivity of the thermopiles over e.g. Bi/Sb solutions. However, it also 
makes that the thermopiles benefit from the reliability, stability over time and 
temperature operating range offered by CMOS technology. 
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Parameter Requirement 



Number of pixels 1 or 2 

Operating temperature range [-40°C , +85°C] 

Typical range of object temperatures T 0 [0°C , +50°C] 
where measurements can be made 1 

Typical range of ambient temperatures T a [-20°C , +85°C] 
where measurements can be made 1 

Object temperature measurement accuracy Error < 1°C for T a within [0°C , +50°C] and T 0 within [15°C , 
in and directly around the comfort range; +40°C] 
valid for objects with emissivity of99.2% 2 . 

Maximum drift on object temperature Lower than 1 °C 
measurement over 10 years of operation 

Aesthetical integration in car interior The thermopile sensor is packaged in a metal TO can. The metal 

cap is not suited to appear in the car interior. The only part of the 
sensor that should be visible is the IR window of the TO cap. 
There are two proven solution: a: TO can with window inside and 
plastic cover hiding the cap, or TO can with window outside. Both 
provide a satisfactory appearance, and have their own application 
in function of installation method and thermal environment. 




Figure 1 : Photograph of thermopile Figure 2: Photograph of 

sensor with window inside. thermopile sensor with 

window outside. 



Automotive qualification Passing automotive testing; 

high-temperature operation; 
temperature/humidity cycling, thermal shock; 
mechanical shock, vibration; 
chemical resistance (cleaning fluids, soft-drinks,...); 
EMC: electromagnetic compatibility, ESD tests; 
Failure rates over 1 0 years of operation Sub-ppm 



Tab. 1. Requirements. 1 Outside these ranges, a temperature overflow or 
underflow signal is generated. 2 See the remark on emissivity in 
chapter 6.2 
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For the signal processing of the thermopile signal, an advanced mixed analog- 
digital signal conditioning MLX90313 ASIC is combined with the thermopile 
infrared sensor in the TO-can. The ASIC is manufactured in standard CMOS 
technology. The amplifier used for the thermopile signal has an offset limited 
to 4 pV, and input referred noise of only 25 nV/Hz 1/2 . 



ESD protection 
and 





power supply 
decoupling 



PCB 



Connector 



Signal conditioning ASIC 



Thermopile sensor in TO-can 



Fig. 3. MLX90601: automotive module assembly with thermopile sensor 

and signal conditioning ASIC 



The high accuracy on the object temperature measurement within a wide 
range of ambient and object temperatures is obtained by making the ambient 
compensation and linearization in the digital domain. Both thermopile and 
thermistor output are amplified and converted to digital. The ambient tem- 
perature is calculated on basis of the amplified thermistor voltage and the 
thermistor calibration constants stored in EEPROM. The object temperature is 
calculated by solving the 4 th order equation 



V TP =a(T 0 4 -T a 4 ) 



(i) 



In this equation, T 0 is the object temperature we want to measure, 7 a is the 
ambient temperature of the sensor itself; V TF is the amplified thermopile volt- 
age and a is the calibration constant determined during object calibration of 
the module in front of a precision black body. 
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The 4 th order equation (1) is solved exactly in the digital domain. As the equa- 
tion is not approximated by a first or second order Taylor approximation 
around a working point, good accuracy is obtained over a wide range of tem- 
peratures, and not only directly around the working point. 

The absolute object temperature accuracy is mainly determined by the toler- 
ance on the temperature of the laboratory bath used during ambient calibra- 
tion, and the tolerance on the temperature of the black body used during 
object calibration. These tolerances lead to small errors on the measured 
ambient temperature 7" a , and on the calibration constant a. Equation (1) shows 
that the measured object temperature T 0 is not impacted by the error on a if 
T 0 = T a (and V TP = 0). As the difference between T 0 and T a increases, the error 
on the measured value of T 0 will also increase. 

The calibration constants are stored in the EEPROM of the signal conditioning 
ASIC. Although the EEPROM has passed testing for 10 years data retention at 
85°C and 1 year at 125°C, extra safety for data retention is built in by apply- 
ing a 5 bit Error Checking and Correction (ECC) code. The ECC is based on 
Hamming coding. Error checking and correction is run after each power up; it 
allows automatic correction of single bit errors. Multiple bit errors lead to an 
error message. 



4.2 Integrated Sensor/ ASIC Solution 

The newest development in the area of automotive passive infrared sensors is 
the integration of the temperature sensing element and the signal condition- 
ing ASIC into a standard 4-pin TO-39 can. 

This infrared thermometer combines the functionality, accuracy, reliability 
and know-how of the current MLX90601 module in a very small package, 
made possible by the design of a reduced size thermopile chip and a small sig- 
nal conditioning chip in standard 0.35 pm CMOS technology. The assembly 
supports both a single and a dual sensor version suitable for dual zone air-con- 
ditioning and blind spot detection applications. The factory calibrated, meas- 
ured temperature is available at the TO-can pins as a standard PWM signal or 
the information can be read using the 2 wire SMBus protocol. The thermome- 
ter supports both 3 V and 5 V supply voltage operation as a standard and 12V 
supply voltage operation for remote installation. 
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To the MAC system manufacturer the advantages are multiple: 

► Standardized factory calibration leads to shorter development time and 
lower product cost. 

► Reduced part count improves reliability further and lowers cost. 

► Integration on customer PCB without extra sensor PCB is possible. 

► Reduced size, space saving in a more and more cramped area as more 
and more applications need to find a place behind the dashboard; 

► Complete single/dual zone thermometer in one package; 



Fig. 4. MLX90614: digital, fully calibrated, infrared thermometer in a TO 



The Mlx90614 is built from 2 chips developed and manufactured specifically 
for this purpose: a reduced size infrared sensor and a signal conditioning ASIC, 
specially designed to process the output of IR sensor. Thanks to a low noise 
amplifier, a high resolution ADC and powerful DSP unit the thermometer can 
achieve a high resolution and accuracy over wide temperature ranges. The 
infrared thermometer is factory calibrated in wide temperature ranges: -40 to 
125°C for ambient temperature and -70 to 380 °C for object temperature. For 
object temperatures in range 0°C to 50 °C the measurement error is below 
0.5°C. Outside that range the error is below 1%. 

The 10-bit PWM is configured to transmit continuously the measured object 
temperature for range of -20 to 120°C, which corresponds to an output reso- 
lution of 0.14°C. The PWM can easily be customized digitally for any temper- 
ature range desired by customer without affecting the internal calibration of 
the device. Using the SMBus, the measured temperature is directly accessible 
with a resolution of 0.01° C. 




can 
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Compared to the MLX90601, the new MLX90614 thermometer in a can has 
some improvements: 

► The user has the opportunity to adopt the sensor for any emissivity 
coefficient below one (see chapter 6.2) without affecting the factory cal- 
ibration of the sensor. 

► The measurement bandwidth and associated noise level can be chosen 
independently from the output refresh rate. By a software implementa- 
tion of an Infinite Impulse Response filter, we can keep the refresh rate 
constant while at the same time achieving a low noise level for the 
measured signal. 

► Using the SMBus protocol, the user can build a sensor network of up to 
10 infrared sensors connected with only 2 wires, each device having its 
unique 8-bit ID. 

The thermometer is available in 2 versions: 5 V compatible and 3 V (battery) 
compatible. Additionally, the 5 V can be easily adopted to operate from high- 
er supply voltage (6-16 V) by adding only one external bipolar transistor, one 
resistor and a capacitor. 



VDD=3 or 5V 




Fig. 5. 3 or 5 V supply voltage version of the MLX90614 




Fig. 6. 12 V supply voltage version of the MLX90614 for remote applica- 

tions 
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5 Possibilities of Infrared Arrays in Automotive Comfort Sensing 

Dual pixel and multi-pixel (array) sensors have the following benefits in com- 
fort applications: 

► Independent sensing of comfort in two or more zones: thereby provid- 
ing a real dual or multi-zone automatic climate control using a single 
sensor with multiple thermopiles integrated. This approach can drasti- 
cally reduce wiring cost, since the mounting location of the senor is 
decoupled from the area that is being measured. Also it is no longer 
necessary to use a sunlight sensor as the effect of sunlight radiation is 
directly captured by the array thermometer; 

► Distinction between skin and clothes: The extra information may allow 
making the control system more intelligent; 

► Filtering of anomalies such as the hot tip of a cigarette: Such anomalies 
can be detected very clearly when using an array as an array is 
equipped with a lens and individual fields of view per pixel can be made 
quite small without compromising the signal to noise ratio. Melexis 
made 4x4 array prototypes with individual field of view of 10°, and 
10x10 array prototypes with individual field of view in the order of 5°; 

The main advantages of an array can be realized by using the same sensor for 
two applications: comfort and safety. This will be discussed in chapter 7. 



6 Practical Aspects of Automotive Comfort Sensing 

6.1 Sunlight Immunity 

Direct sunlight incidence in the thermopile sensor should not impact the accu- 
racy of the object temperature reading. To this end, the sensor window filters 
out the short wavelengths below 5.5 pm. Long wavelength infrared radiation 
emitted by the objects in the car will pass the filter window. Long wavelength 
infrared radiation emitted by the sun is blocked by the windows of the car 
(glass is not transparent for long wavelength infrared radiation!). The com- 
bined filtering of the Si-window of the sensor and the glass of the windows of 
the car reduces the impact of direct sunlight at orthogonal incidence to an 
error of +0.5°C to +1°C on measured object temperature. The error can be 
further reduced by mounting the sensor in such a way that direct orthogonal 
illumination by the sun is impossible or rare. 

If windows are open, long wavelengths emitted by the sun are NOT blocked 
by glass, and direct sunlight incident on the sensor can lead to important 
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changes in object temperature readout. This should however not be for the 
climate control application, as it is normally not designed for functioning with 
windows open. 



6.2 Emissivity 

Emissivity is the ratio of power radiated by a surface to the power radiated by 
a blackbody at the same temperature. Melexis infrared sensor modules are cal- 
ibrated in front of a black body with emissivity 99.2%. That means that the 
calibration constant a is determined in such a way that the correct object tem- 
perature is obtained when measuring objects having an emissivity of 99.2%. 
Typical surfaces in a car have somewhat lower emissivities [8]: skin (0.97 to 
0.98), glass (0.92), black plastic (0.95), textile fabrics (Hessian fabric, green 
coloured, reported 0.9). 

For simplicity, we further assume that the calibration black body has emissiv- 
ity = 1. In that case, we can take into account the emissivity of the object by 
changing equation [1] into: 



V TP =a (eC-t) 



(2) 



Where £ is the emissivity of the object. 

However, equation (2) is not completely correct for following reason. An object 
with infrared emissivity e reflects l-£ : the lower the emissivity, the higher the 
reflection of infrared radiation. So, the sensor will also see radiation from the 
surroundings/environment reflected by the directly observed objects. In a first 
approximation, we assume the surroundings in the comfort application to be 
at a uniform ambient temperature T a . In that case, a more correct formula tak- 
ing into account the infrared radiation reflected by the object is: 



v; p =a (er o 4 + (1 -e )T 4 a - T a 4 ) = as (T a 4 - T 4 a ) 



(3) 



The simplified model of formula (3) shows that the measured thermopile volt- 
age is a factor £ smaller than what it should be for a correct measurement, 
which results in a too small difference between the measured object tempera- 
ture and the ambient temperature. If the object has emissivity lower than 
99.2% the following will happen: an object at higher temperature than ambi- 
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ent T a will be measured too low; an object at lower temperature than T a will 
be measured too high. The importance of the error grows with the difference 
between T 0 and T a , and with the value of the emissivity: the lower the value 
of the emissivity the larger the error. The phenomena predicted by the sim- 
plified model (3) have been experimentally confirmed by Melexis. The order of 
magnitude of the error can be easily derived from (1) and (3). For T a = 45 °C 
and T 0 = 20 °C, and £ = 0.95 (instead of 0.992), the measured T 0 using formu- 
la (1) will be about 21.2°C / or a positive error of 1.2 °C. This error should not 
prohibit the use of the sensors in automotive climate control: the effect can be 
taken into account in the design of the climate control system, by checking out 
typical operating conditions. If needed, the sensor ASIC can easily take into 
account the effect of a different emissivity without affecting the calibration. 
The processor of the climate control system can make a compensation on basis 
of both T 0 and T a . The main sensor characteristic of importance for the climate 
control application is the stability and repeatability of the temperature read- 
ing. 



6.3 Field of View 

The temperature an infrared sensor measure is an average of the temperature 
of all surfaces the sensor sees in its field of view. The field of view required 
for a comfort application is determined by the vehicle dimensions and the 
design-principles of the manufacturer of the air conditioning control panel. As 
a thermopile die is sensitive to radiation coming from all angles (180° front 
view; even sensitivity to radiation coming from the back of the die), it is not 
difficult to make thermopile sensors with wide field of view. Reducing the field 
of view is possible by reducing the diameter of the opening of the TO-can. 
However, this technique has its limits. At a certain moment, the amount of 
energy received from the object becomes too low, and instabilities occur. 
Typically, it is advised to work with a viewing angle of at least 60°. Smaller 
field of views can be realized through lenses, but these make the application 
considerably more expensive. 

A special case is the dual thermopile sensor. In the sensor are two thermopile 
chips which are placed off centre. Fig. 7 shows a typical field of view for such 
a sensor. Each of the thermopiles sees a different area of the car such that one 
looks at the driver and his/her surroundings and the other looks at the front 
passenger and his/her surroundings. Both the temperature of the driver side 
and the passenger side can be measured separately and transmitted to the air 
conditioning controller. This makes this sensor especially suited for multi zone 
car air conditioning. 
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0 ° 



Fig. 7. typical horizontal field of view of a dual thermopile infrared sensor 



6.4 Thermal Gradients 

An infrared thermometer module is calibrated in front of a reference black 
body (perfect thermal emitter with £=1). During the calibration, the module 
itself is operated at room temperature, and the module is isothermal: the dif- 
ferences in temperature over the surface of the module are very small. When 
used for measurements under isothermal conditions, the measurement results 
will be correct. However, when used under non-isothermal conditions, errors 
may occur. A typical automotive example is the example according to which 
the module is mounted in the dashboard. The back of the module is behind the 
dashboard and the thermopile sensor looks into the cabin through an opening 
in the dashboard. High-power electronics mounted close to the module heat 
the back of the module; the front part of the sensor is cooled by the air-condi- 
tioning. In such situation, the top of the sensor package has a slightly differ- 
ent temperature than the thermopile chip, and emits radiation according to its 
temperature. This infrared radiation is an error signal added up to the infrared 
radiation emitted by the object. The error can become important when the 
field of view is small, and radiation received from the object is low. 

This problem was solved by providing thermopile sensors in metal cans of 
highly thermally conductive material; so the thermal gradient over the sensor 
is kept small. For demanding applications where the sensor has to be used 
under high thermal gradient, the thermopile sensor of Fig. 2 has best per- 
formance, among others thanks to its thick aluminium cap. For stable opera- 
tion, it is always advisable to limit the thermal gradients over the thermopile 
sensor by avoiding placing of the sensor close to strong heat sources. 
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6.5 Cigarette Smoking 

A property of IR radiation is that it is almost not influenced by smoke in the 
air. Smoke from cigarettes will therefore in all practical circumstances not 
influence the temperature reading of the IR sensor. The temperature of the 
tip of a lit cigarette can reach up to 800K. The size of the tip is very small how- 
ever and when the cigarette is not too close to the sensor ( > 50 cm) the tem- 
perature readout influence will stay below 0.5°C (measurement with a wide- 
angle sensor with an opening angle of about 90°). When the cigarette is 
drawn the temperature of the tip can reach 1000K and the readout may react 
up to 1 °C. Normally this is only a very short effect which can be averaged out 
easily by the MAC controller or by setting the sensor to a lower bandwidth. 



7 Safety Applications 

7.1 Windshield Defogging based on Dew-Point Detection 

Windshield fogging presents a double safety issue, as it deteriorates the view 
on the road, and it distracts the driver who is looking for manual activation of 
the defogging. Reference [6] describes how the fogging of the front windshield 
can be measured and predicted by using an infrared sensor for measurement 
of the temperature of the windshield. According to [6], the infrared sensor can 
simultaneously be used to make a highly efficient climate control system. This 
can lead to considerable cost savings as expensive and difficult to install con- 
tact sensors can be avoided. 

Infrared comfort sensors used in air conditioning control systems can be 
directly used in windshield defogging applications. The operating temperature 
ranges are equal; there are no special accuracy requirements. The require- 
ments related to automotive reliability are equal to the ones used in automo- 
tive comfort sensors. 



7.2 Blind Spot Detection 

System designers are examining different types of sensors to help drivers be 
aware of vehicles in their side blind spots when changing lanes and making 
turns. Dual infrared sensors integrated in mirrors, taillights or side fascia can 
measure adjacent lane temperature over time to detect vehicles entering the 
side blind spot. Such systems ignore stationary objects, minimizing false 
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alarms and are immune to noise. Dual infrared comfort sensors used in MAC 
systems can also be used for blind spot detection. 



7.3 C0 2 Concentration Measurement 

Future climate control systems in Europe will most likely use C0 2 as a cooling 
agent. In these systems, C0 2 will be used under high pressure. Several studies 
on the safety and health risks of C0 2 in automotive cooling/heating systems 
are being made. According to the study [7] made by the EPA (Environmental 
Protection Agency USA), it is strongly recommended to use C0 2 detection 
inside automotive climate control systems in order to activate system fault 
indicators, and to automatically activate fast release of the C0 2 cooling agent 
and air exchange in the passenger cabin. The open air C0 2 concentration is 
365 volume-ppm. Sustained exposure to 40.000 volume ppm (1 hour) is con- 
sidered as harmful, as well as short exposure to 60.000 volume ppm (minutes): 
see [7]. The infrared absorption line of C0 2 around 4.26 pm can be used to 
make accurate and reliable C0 2 concentration measurements. 

C0 2 concentration measurements require the use of thermopile sensors with a 
filter window centered on the C0 2 absorption wavelength of 4.26 micrometer. 
The concentration measurement is based on the use of an infrared radiation 
source having strong emittance around 4.26 pm. Reliable sources operating at 
a temperature around 402 °C ( = black body temperature with peak radiation 
4.26 pm) are commercially available. The radiation of a black body at 402 °C is 
strong enough to emit a power level in the narrow absorption line around 4.26 
pm that is of the same order of magnitude as the power emitted by a black 
body at room temperature over the whole width of the spectrum, thus bring- 
ing the module in its normal operating range. 



7.4 Presence Sensing, Seat Occupancy and Pedestrian Detection 

The passenger airbag should not be activated if a passenger seat is not occu- 
pied. An infrared array can be used to detect the presence of a person in a seat 
by detecting differences in measured temperature among pixels. 

Intelligent airbags need to do more than just detect whether a seat is occupied. 
The activation of an intelligent airbag depends upon the type of person (baby 
in seat, child or adult) and the position of the person (sitting straight, bent 
over, feet on dashboard). As such, an intelligent airbag needs one or more sen- 
sors to distinguish among the different situations. Thermopile-based infrared 
sensor arrays can make rough thermal images, which allow to learn about the 
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type of person and to detect the position of a person using minimal image pro- 
cessing. 

Infrared sensor arrays can also be used to detect obstacles outside the vehicle. 
The advantage of using infrared above visible detection is in the extra dimen- 
sion provided by the temperature information. Thanks to this extra informa- 
tion, image processing power and complexity can be greatly reduced, and the 
number of erroneous detections can be strongly reduced. A visible camera 
with image processing system might have difficulty in distinguishing a real 
person from its shadow; for an infrared camera, such problem does not exist. 

Presence sensing, seat occupancy and detection of pedestrians all require 
arrays. They are based on the detection of differences in temperature between 
the surface of a person and its environment. Three parameters of the arrays 
are of importance for the applications: 

► Spatial resolution, determined by the number of pixels; 

► Temperature resolution: the smallest temperature difference distin- 
guishable between two pixels; 

► Detection speed; 

Sensing the presence of a person in a car seat is the simplest application of the 
three; a limited number of pixels will be enough to determine whether a per- 
son is present in the seat or not. Typically, a 4x4 thermopile array should be 
sufficient. 

In order to sense the type of person (child, adult) and the position of the per- 
son, more information is needed. A minimum of 10x10 pixels is advised. 

In order to detect a person at sufficient distance to stop (25 m braking distance 
at 60 km/h on a wet road), we assume that the width of the person (0.5 m) 
should fill a single pixel, and that we scan the road over a width of 10 m and 
a height of 3 m. These assumptions lead us to the need for a minimal horizon- 
tal resolution of 22 pixels and a minimal vertical resolution of 7 pixels, leading 
to a 1 54 pixel array. 

During a nice summer day, the surface temperature of the seats can get quite 
hot. In such situations, the temperature difference between the skin of the per- 
son and the surface of the seat can become small. Most probably, it will not be 
possible to make a failure-free detection system. In some cases, the presence 
of a person will not be detected, as the difference in temperature between the 
person and its environment is not clear enough. An infrared array can only be 
an aid in detecting, but cannot faultlessly cover 100% of the practical situa- 
tions. In order to reduce the number of detection failures, it is important to 
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have a good temperature resolution and stable readout, typically better than 
1°C or even 0.5°C. 

Presence sensing and seat occupancy can deal with quite low frame refresh 
rates: changes in presence and changes in position of a person do not happen 
within fractions of seconds. 

Pedestrian detection however, needs quite fast frame rates. At 60 km/h, a car 
moves 17 m every second. If the frame refresh rate is above 1 s, a pedestrian 
detection at 25 m is of no use. To make detection at 25 meter distance useful, 
the frame should be refreshed within about 1 meter travelling distance, or 
within 60 ms, for a vehicle travelling 60 km/h. 




Fig. 8. 90611 Block diagram 



In order to yield good temperature resolution, infrared sensors need some inte- 
gration time. A larger integration time results in lower noise and better tem- 
perature resolution. So, speed is a requirement that is in conflict with temper- 
ature resolution. 

We have made a 4x4 prototype infrared array module MLX90611 which can 
be used in presence detection and seat occupancy by developing a thermopile 
sensor die with integrated multiplexer electronics, as well as the infrared sig- 
nal conditioning ASIC MLX90321, that is capable of handling up to 32 pixels 
(Fig. 8). 

The MLX90321 reads out 4 pixels simultaneously, thereby increasing the 
frame refresh time considerably. A key feature of the MLX90321 is its ability 
to communicate through virtually any protocol (up to 4-wires) with external 
sensors, actuators or other devices, as all SPI pins can be reconfigured as gen- 
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eral purpose input-outputs. The data from these devices can be processed by 
the MLX90321, combined with infrared temperature measurements and sent 
back to the same or other devices. This option provides flexible control of the 
module as well as economical (in terms of wiring) and safe transfer of data. 

The MLX90611 IR array module has a 4x4 array sensor with wide viewing 
angle. The total field of view of the complete array is 70 by 70 degrees, each 
pixel having a field of view of 10 degrees. Each individual pixel has its own 
constants stored in the ASIC EEPROM and is fully calibrated by Melexis. The 
temperature information of each individual pixel can be communicated to the 
ECU through an SPI or LIN interface. 

The 4x4 array prototype can be used as a combined sensor for comfort sens- 
ing and presence detection. The photograph in Fig. 9 shows a picture of the 
4x4 array module, while Fig. 10 gives a schematical drawing of the field of 
view of the module. 




Fig. 9. 4x4 array prototype module; 

thermopile array in TO-39 
package with electronics 
on PCB. 




Fig. 10. Field of view of 4x4 
array prototype. 



In addition, Melexis developed a 10x10 thermopile array module. This proto- 
type is based on the MLX90313 signal conditioning ASIC, which reads out all 
pixels sequentially. An example of possible application of the 10x10 array is 
shown in Fig. 11. The figure shows an overlay of a low-resolution infrared pic- 
ture taken with the 10x10 array, and a photograph taken in the visual domain. 
The right photograph shows a situation with the driver sitting upright; in that 
case, the airbag can blow off at full energy. The left photograph shows a situ- 
ation where the driver grabs a map lying under the windshield. The chest of 
the driver is against the steering wheel. In that position, the airbag should not 
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blow off. The two photographs show that the position of the driver can be eas- 
ily detected by identification of the hottest spots in the 10x10 matrix. As there 
are only 100 pixels, image processing requirements are very limited, while the 
probability of correct detection is rather high. It should be clear that the 10x10 
array can at the same time be used for comfort sensing. 





Fig. 11. Application of 10x10 infrared arrays for seat occupancy / position 
detection 



At this moment, no prototypes suitable for pedestrian detection are available. 
Although the application can probably be realized using thermopile array 
technology, the high frame renewal rate combined with the good temperature 
resolution will require massively parallel processing of pixels, which might 
become only commercially viable using future microelectronics technology 
generations. 



8 Conclusion 

In this paper, an overview has been given of thermopile based infrared sensor 
applications in automotive comfort and safety. For each of the different appli- 
cations, the sensor requirements put have been discussed and a number of 
practical considerations related to the practical use of infrared sensors in an 
automotive environment have been discussed. 

The application of infrared thermopile sensors as the comfort sensor for air 
conditioning systems in several car models has shown that this is a mature, 
reliable technology. Further advances in integration will result in an even 
more versatile technology with widespread applications. The cost reduction 
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coming from the higher integration level will make the infrared thermometer 

cost competitive with other automotive cabin temperature sensors. 
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Abstract 

Hand gestures as a natural form for communication have attracted 
many research efforts. In this paper we present a hand-gesture con- 
trol system for drivers or passengers based on a new generation of 
a 3D imaging sensor, which generates both a grey image and a 
range image of the scene. In this gesture recognition system, we 
take the hand as one entity, and only the 3D position or the trajec- 
tory of the hand centroid is interesting for the system. The advan- 
tages of the proposed system over other gesture recognition tech- 
niques are its robust and real-time recognition, effective suppression 
of background illumination, and no need of additional devices or 
markers to be attached. As a driver assistance system, the proposed 
gesture-recognition system is natural, robust, efficient and practical. 



1 Introduction 

With the rising number of intelligent driver-assistance functions emerging in 
modern car architectures, it becomes more and more important to look for 
more natural, comfortable and efficient Human Vehicle Interfaces. Gesture 
recognition provides a new channel of input and enhances usabilities. 

Hand gestures as a natural form for communication have attracted many 
research efforts [3, 10]. Many researchers focus their work on the analysis and 
recognition of the articulated hand motion. Some use reflective markers or 
gloved-based devices with e.g. electromagnetic sensors [5]. These systems are 
cumbersome, intrusive or expensive. Efforts were also made to track motion of 
bare hand [11], however, which are either limited by user-specific hand mod- 
els or by well-designed background to make hand segmentation easy. Others 
work on the combination of two or more modalities, e.g. speech and gesture 
[8], in order to maximize the usability. 

In this paper we present a hand-gesture control system for drivers or passen- 
gers based on a new generation of 3D imaging sensor. The sensor developed 
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by PMD technology GmbH is based on the principle of Photonic Mixer Device 
(PMD) [12]. It generates a dense range image of the scene at video rates by 
analyzing the phase shift between a modulated infrared signal emitted by a 
LED array and the reflected signal captured by the sensor. The phase shift is 
caused by the time of flight of the light signal from the light source to an 
object point and backwards to the sensor as shown in Fig. 1. 




Fig. 1. The PMD Principle: A PMD system consists of a combined illumi- 
nation unit and a sensor unit. A modulated infrared signal emitted 
by an LED array travels from the light source to an object, where 
it is reflected, and then travels back to the sensor. Due to the time 
of flight, a phase shift occurs between the emitted and the sensed 
signal. Based on this phase shift, the distance between the PMD 
system and the object is calculated. 

The production process of the PMD sensor is similar to that of a standard low- 
cost CMOS imaging sensor. Therefore, the complete system can be produced 
at a very low price on embedded platforms. 

In our gesture control system, we take the hand as one entity, and only the 3D 
position or the trajectory of the hand centroid is interesting for the system. 
The advantages of the proposed system over other gesture recognition tech- 
niques are its robust and real-time recognition, effective suppression of back- 
ground illumination, and no need of additional devices or markers to be 
attached. 

In the proposed system, the 3D sensor is used to capture the motion of the dri- 
ver's or the passenger's hand in real time. Through processing the range 
image obtained from the 3D sensor, the hand position is estimated within the 
camera's field of view. An object tracking method, based on the Kalman filter, 
is used to stabilize the hand-tracking performance, even under rough condi- 
tions. We included two modes of control in the system. In the first mode, the 
estimated 3D hand position functions similar to a 3D computer mouse. In the 
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second mode of control, movement patterns of the hand are classified with a 
set of pre-defined/trained Hidden Makov Models in order to recognize complex 
gestures. Both modes can serve to press buttons, select menu items, scroll win- 
dows or lists, or move or zoom maps in a navigation view. 

The presented system architecture consists of a PMD camera and a separate 
processing unit for analyzing the 3D data. The imaging and the image-pro- 
cessing technologies presented in this paper can be extended and perfectly 
used for applications like out-of-position sensing and occupant classification. 

In the following section we present our hand-gesture control system in detail. 
We report our experiment results in section 3 and at last conclude the paper 
in section 4. 



2 The Proposed Gesture Recognition System 

In this section we present a hand-gesture control system based on a 3D imag- 
ing sensor. The hand motion of the driver or a passenger is captured by the 
sensor in real time. Through analyzing the image data provided by the PMD 
camera, we can estimate the 3D position of the hand within the camera's field 
of view. An Kalman-filter tracking method is employed to stabilize the hand- 
tracking performance. 

In the following we first describe the algorithm of hand position detection, and 
then propose a Kalman-filter model for hand tracking. Afterwards two control 
modes are presented for monitoring the system. 



2.1 Hand Detection 

To determine the hand position we need to segment the 3D hand points from 
the background. From the range image and the extrinsic and intrinsic param- 
eters of the PMD camera we can compute directly the corresponding 3D points 
for every pixel on the image. But not all the 3D points in the field of the cam- 
era's view are of interest for us to estimate the hand position, since the hand 
of the driver or the passenger is supposed to be placed only within a certain 
3D volume, when it "monitors" the system. Therefore, we define a 3D-volume 
of interest where the hand is expected to move, and the process of hand seg- 
mentation works only on the 3D points within this volume. We name this 3D 
volume action volume. Fig. 2 shows a possible setup of the camera's view and 
the action volume. Note that the system is easy to handle and no calibration is 
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needed. The camera can be positioned more or less anywhere. The action vol- 
ume can be arbitrarily adjusted within the field of the camera's view. 




Fig. 2. An example setup of the camera's view and the action volume 

The following five steps are implemented to detect the hand points and esti- 
mate the hand position. Fig. 3 gives an overview of the complete hand-track- 
ing process. 

Smooth the raw data 

In order to enhance the quality of range data, we apply a standard mean filter 
on the sensor's raw data. According to our practice, smoothing upon the raw 
data obtains better result than smoothing upon the range image. 

Remove out-of-range 3D points 

As mentioned above, only the 3D points within the pre-defined action volume 
are of interest for hand detection. Moreover, in the next section we will 
describe a Kalman-filter model used for hand tracking. The Kalman filter can 
give a prediction of the hand position in the next frame. Hence, we define a 
certain neighborhood around the predicted position, and search for the hand 
points within the intersected volume of this neighborhood and the action vol- 
ume. 

Remove noisy points 

The quality of the range measurement directly depends on the fraction of the 
emitted light being reflected by an object. The lower the fraction is, the lower 
the reliability of the depth measurement is. Therefore, we remove points with 
low fraction values based on a variable threshold. 
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Remove edge-points 

Additionally, to segment the hand from the scene we are interested actually 
only in points on planes instead of edges. With respect to 8-neighbour rela- 
tionship, we define the 3D points with less than two near neighbours as edge- 
points. Near neighbour here means the Euclidean distance between two neigh- 
bour points is less than a certain threshold. 



Segment the hand points and compute the centroid 

We define a 3D front direction for the action volume and sort the 3D points 
descendingly along this direction. We take a certain number of the front points 
as the hand points, and compute their centroid as the estimate of the hand 
position. In this operation we assume, the hand points are always in the front 
of the action volume. 




Fig. 3. Diagram of the hand-tracking process 



2.2 Hand Tracking Using the Kalman Filter 

In the previous section, we presented an algorithm for detecting the hand 
position from a single PMD frame through analyzing its image data. The main 
idea is to assume the hand points to be the front points within the action vol- 
ume. However, due to noise and disturbance, this assumption may not always 
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be true. In this section we propose a Kalman filter model, which not only 
smoothes the motion of the hand but also deliver a prediction of the hand posi- 
tion in the succeeding frame. 

The Kalman Filter tries to deduce an optimal estimate of the past, present, and 
future state of a linear system using a time sequence of measurements, given 
a statistical model of the system and an initial estimate of the state [9]. 

In our hand-tracking problem the state x k is the true 3D hand position at the 
time k. The measurement z k is the estimated hand position as presented in the 
last section. Our Kalman filter model for hand tracking is based on the assump- 
tion of the relatively consistent acceleration of the hand motion, which yields 
the equations 



(x k - x k _ x ) - (X k _! - x k . 2 ) = (x k l - x k . 2 ) - (X k _ 2 - X k _ 3 ) + W VA , 
i.e. x k = 3x k _! - 3x k _ 2 + x k . 3 + w k4 

and z k = x k + v v . 



where w k is the process noise and v k is the measurement noise. We assume 
both of them follow the isotropic Gaussian distribution. The complete process 
of the Kalman filtering is composed of the iterations of two main operations: 
prediction and correction. For detailed derivation of the Kalman filter please 
refer to the Kalman filter tutorial [9]. 



In this system we use the hand-detection method as stated in the previous sec- 
tion, to estimate the hand position at the time t= 0 and t= 1. Then apply the 
above Kalman-filter method to the succeeding time. The intial estimate of the 
state x 2 at time t= 2 is the result of the hand-detection method. The Kalman fil- 
ter is applied to "correct" the estimate and deliver a prediction for the next 
moment. As mentioned in Section 2.1, the prediction from the Kalman filter is 
used to guide the detection of the hand points in the next frame. Please see 
Fig. 3 for the complete process of hand tracking. 



2.3 Hand-Control Modes 

Two hand control modes are implemented in our system. In the first mode, the 
estimated 3D hand position functions similar to a 3D computer mouse. In the 
second mode, movement patterns of the hand motion are classified with a sta- 
tistical model in order to recognize complex gestures. Both are used to press 
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virtual buttons, scroll within selection lists or to move or zoom maps in a nav- 
igation menu. 



2.3.1 3D Hand-Mouse Control 

In the 3D hand-mouse control mode, the auto driver or passenger can use 
his/her hand to control the mouse-icon on the screen. It behaves like a 3D com- 
puter mouse. The plane of X- and Y-axes in the 3D action volume resembles 
the 2D surface of the screen; and the Z coordinate of the hand position decides 
whether or not to press a button, select a menu item, or hold the cursor to shift 
a map or to scroll a window. 

However, not like the "real" mouse that we hold on a flat table, the hand mouse 
vibrates in the 3D volume, due to noises and the natural vibration of the 
human hand motion. Accordingly, the mouse-icon will be flickering significant 
on the screen, if the X- and Y-coordinate of the 3D hand position reflects direct- 
ly the mouse position. Therefore, we also implemented an anti-flickering 
process to stabilize the displayed mouse icon. The basic idea is to update the 
mouse position only when the next mouse position is outside a certain size of 
the local neighborhood around the current mouse position. 

There are several ways to realize the pressing of a button. One possibility is to 
interpret any significant forward movement of the hand as a button press. 
Another possibility is to take the absolute distance to a fixed display as the cri- 
terion. Currently we are working on the interpretation of the hand posture 
(e.g. "opened" and "closed") in order to solve this task. 



2.3.2 Gesture- Pattern Control 

The second mode is intended to control functionality of the client application 
only by moving the hand according to previously defined or trained movement 
patterns. Each item of this gesture vocabulary corresponds to a gesture com- 
mand which can be performed if a movement pattern is recognized by the sys- 
tem. Typical gestures that can be modelled are commands like "select", 
"back", "play", "pause" or "zoom in" and "zoom out". A typical movement pat- 
tern for the zoom commands can for example be a circular motion of the hand 
performed clockwise or counter clockwise in order to distinguish between 
"zoom in" and "zoom out". 

The big advantage of this approach compared with the 3D hand-mouse mode 
is that the passenger has seldom to have a look at the screen (for example only 
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to receive the results of the navigation system), because for most commands 
no steering of a cursor is needed. Hence the driver is much less distracted in 
this mode. 

We use Hidden Markov Models trained on a previously recorded training set 
of gestures to classify observed movement patterns. Hidden Markov Models 
(HMMs) are stochastic models used in many pattern recognition systems. They 
are commonly applied in speech recognition systems. There they are usually 
used to model time consecutive feature vectors extracted from speech signals 
[7]. Hence it makes sense to use them in our gesture recognition system where 
the movement patterns of the hand have to be classified which are also time 
consecutive. 

Hidden Markov Models have already been used in other gesture recognition 
systems [6]. Our approach differs from the existing methods in that point that 
we calculate our features based on real 3D hand positions which are the 
results of the segmentation step described earlier in this paper. 

The simplest way of coding a feature sequence belonging to a gesture is to use 
the sequence of the raw 3D positions o = o 1# ..., o k as a feature sequence 
f = f lr ..., f n as shown in Fig. 4. Advanced representations of f also used by 
other researchers, such as the cartesian velocity (< dx , dy, dz), the polar velocity 
with angular velocity term (dr, d(j), dz), the polar velocity with tangential veloc- 
ity term (dr, rd(j), dz) or other comparable features [2]. 

In our system, we train individual models HMMj , with / = 1, 2, ..., N, for each 
gesture command out of a set of N gestures (e.g. the commands mentioned in 
the beginning of this section) with a large number of training samples. The 
parameters of each model HMMj are estimated using the Baum-Welch Re-esti- 
mation procedure [1]. 

After finishing the training procedure, we can use the HMM models to classi- 
fy previously un-trained movement patterns represented by their correspon- 
ding feature sequences f. According to the Bayes' Theorem, the optimum clas- 
sifier decides for the class with the maximum a posteriori probability 

argmax i fP(Ci I f)} with i = 1,2 N 

for a set of classes C ; . The rule of Bayes yields 
P(c \ I f ) = P(f I C ; ) * P(C ; ) / P(f). 
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Looking for the maximum value, the probability of occurrence P(f) can be 
skipped because it is constant for all classes, and hence only the product of the 
a priori probability P(C J and the class conditional probability P(f \ CJ is left. 

Using our HMM approach, the conditional probabilities for a set of TV gestures 
are given by the probabilities P x (f | HMM ly ), ..., P N (f | HMM N J. The values of 
the probabilities can easily calculated with the Viterbi Algorithm [4]. The val- 
ues of the conditional probabilities give an indication of the similarity between 
the observed movement pattern represented by the feature sequence f and the 
training sequences of each model. 

In a gesture recognition system the a priori probability P(f) strongly depends 
on the current state of the system. For example in "CD player mode", the a pri- 
ori probability for the class "play" should be higher than for the class "zoom 
in". In our current work, we assume the same a priori probability for all ges- 
tures. In this case argmax i { P(C X | f)} is reduced to argmax i {P(f | HMM iy )/. 

Implicitly we decide for the model - in other words for the gesture - with the 
biggest similarity to the tested sequence f . Fig. 4 shows the entire recognition 
process for a set of N trained models. 




argmaxPi . 

Pz(f\HMMo) ie{ 1 N} 

P N (f\HMM N ) 



Fig. 4. Gesture recognition with HMMs 
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3 Experiments 

The presented system architecture consists of a PMD camera and a separate 
processing unit to perform the computation of hand tracking and gesture 
recognition. We used two types of PMD sensors: PhotonICs® PMD lk-S and 
PhotonICs® PMD 19k, with the resolution of 64x16 and 160x120 pixels, 
respectively We chose a frame rate of around 20 fps. 

Fig. 5 shows a screenshot of a sample user interface being controlled by our 
gesture recognition system. In this case we modelled a "touchless touch- 
screen" device. The yellow "hand" in the image is the icon for the hand-mouse. 
In the 3D-hand-mouse control mode, a user can use his/her hand to control the 
hand icon on the screen. When the user's hand moves forward within the 
action volume, the hand icon will get "held", which means the according tar- 
get on the sreen is chosen. For example the "hand" could hold the city map in 
the navigation window, and then move it in arbitrary directions. 

As stated in Section 2, we set several thresholds for detecting the hand points 
and tracking the hand position, e.g. the number of hand points, or the size of 
the neighbourhood around the predicted hand position by Kalman tracking. 
These thresholds are pre-definited for the system according to our practices. 
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Fig. 5. A sample user interface controlled by our gesture recognition sys- 
tem 

Tab. 1 gives the statistical evaluation of the performance of the PMD sensors 
and the processing time for tracking the hand in one frame. As we have men- 





Gesture Recognition Using Novel Efficient and Robust 3D Image Processing I 291 



tioned above, the processing unit for computation is separate from the PMD 
camera. That means the computation process and the image-capture process 
can be run simultaneously. Therefore, in this case, the whole processing time 
of one frame just equals the time for capturing the images in the 3D hand- 
mouse control mode. 



PMD sensors 


Image Capture 
(ms) 


Hand Tracking 
(ms) 


PMD [Vision] ® lk-S 
(64x16 pixels) 


21 


17 


PMD [Vision ] ® 19k 
(1 60x120 pixels) 


47 


46 



Tab. 1. Performance of PMD sensors and the computation to process one 
frame 



4 Conclusions 

In this paper we present a hand-gesture control system for drivers or passen- 
gers based on a new generation of 3D imaging sensor, which can generate a 
range image of the scene at video rates. In the system, the 3D sensor is used 
to capture the motion of the driver's or the passenger's hand in real time. A 
Kalman filter model is created to stabilize the hand tracking. Additionally, two 
control modes are introduced in the paper. Both modes can be used to press 
buttons, select menu items, scroll windows or lists, or move maps in the navi- 
gation view. In the first mode the estimated 3D hand position is used to model 
a 3D mouse as input device. In the second mode complex movement patterns 
are classified with Hidden Markov Models. The advantages of the proposed 
system over other gesture recognition techniques are its robust and real-time 
recognition, effective suppression of background illumination, and no need of 
additional devices or markers to be attached. 

The presented system architecture consists of a PMD sensor and a separate 
processing unit for the evaluation of the 3D data. The complete system can be 
produced at a very low price. For the first time a robust application of 3D 
machine vision can be realized on embedded platforms. The technology pre- 
sented in this paper can be extended and perfectly used for applications like 
out-of-position sensing and occupant classification. 
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Network of Excellence HUMANIST - Human Centred Design 
for Information Society Technologies INRETS & ERT 

A. Pauzie, French National Institute for Research in Transport and Safety 

Abstract 

HUMANIST in a Network of Excellence gathering the main research 
institutions from 15 European countries. The partners involved in 
this network are working in the area of new technologies in 
Transport:, with a specific concern on human centred design, to 
ensure matching between provided systems and services, and users 
needs and requirements, in addition to concern with road safety 
improvement. 



1 Introduction 

In the coming years, the development of the new technologies of information 
and communication is going to transform deeply the uses and the practices in 
transport. In the field of road telematics and driver assistance systems, the 
current developments can constitute a real opportunity of help for the mobili- 
ty and a real improvement of the road safety. Nevertheless, implementation of 
ITS raises numerous questions about acceptability by drivers and about possi- 
ble modifications of behaviour or attitudes. 

The humans factors competencies exist in Europe but are scattered. To obtain 
effective results, it is necessary to integrate the research capacities in Europe. 

For this purpose, a panel of research institutes, members of the European 
Conference of surface Transport Research Institutes (ECTRI) and the Forum of 
European Road Safety research Institutes (FERSI) joined to build up a network 
of excellence HUMANIST aiming at federating the researches in the domain 
of user/system interactions and their applications on in-vehicle information 
systems and advanced driver assistance systems. The aims of this network of 
excellence is to federate the researches in the domain of user/system interac- 
tions and design, and their applications on road telematics and driver assis- 
tance systems. 
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2 Context 

If the driving task has little evolved since the creation of the motor vehicle, this 
situation is changing today under the combined effect of the widespread of 
driver information and communication systems implemented in-vehicle and 
the emergence of advanced driver assistance systems. 

These various systems propose or will propose a certain number of functions 
to the driver, with the objective to facilitate driving task and to improve safe- 
ty of travelling. For example, the access to navigation information allows a low- 
ering of the attentional level involved in orientation process of the driving sit- 
uation. The diffusion of traffic information in real time can be at the origin of 
critical situations avoidance. The messages of alert, concerning road or mete- 
orological events arisen downstream, and diffused as quickly as possible to the 
driver, allow the activation of anticipation process. The adaptive cruise control, 
while maintaining a safe headway with the ahead car, decreases the drivers' 
stress and mental load. Finally, in direct connection with objectives of road 
safety, the active assistance systems conceived specifically to take effect in 
accidental situations, can balance some reaction latencies and decision uncer- 
tainties, inherent to the human functioning in driving situation. 

But the effective realisation of the expected benefits will depend on conditions 
of systems design and implementation: in particular, in which measure the 
system responds to drivers needs, is compatible with their functional capaci- 
ties whatever their age and satisfies the criteria of relevance, usability and 
acceptability. 

Finally, the emergence of automation technologies, that is assistance systems 
being able to take care of some control tasks traditionally assigned to the driv- 
er, brings the problem of the tasks dispatching between human and machine, 
as well as the choice of the logic used for the management of this control shar- 
ing, substitute or co-operative. Particularly in the case of co-operative assis- 
tance, further researches are needed regarding the emergent human driving 
tasks of system supervision, system status awareness, and system response 
representation. 

All this argues for a more active participation of the Human Sciences in the 
various stages of systems conception and for a concept of technological devel- 
opment determinedly centred on Human, in which the assistance is designed 
according to the needs and the capabilities of the human being and not driv- 
en by the technological offer. When, in the end, the systems will be able to 
modulate its assistance according to the analysis of the driving situation 
requirements, one can then speak about cognitive engineering, where a cor- 
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rect matching between the system design and the specificity of human being 
is ensured, for positive consequences in terms of road safety 

The area of road telematics and driving assistance systems is characterised by 
a fast technological development and a strong competition in the industrial 
world between Europe, Japan and USA. The success of the deployment of 
these new products needs, beyond technological research efforts, strong and 
continuous actions in the domains of human factors to guarantee the usability 
and the acceptability of these systems and consequently a real improvement 
of the safety and the mobility 

In addition to federate the research activities in this area, HUMANIST, like the 
others NoEs, has the objective to progress towards the creation of a Research 
European Virtual Centre. The network has built a joint program of research, 
integrating, spreading and management activities with the purpose of sus- 
tainability. 



3 Joint Research Activities 

In the area of human factors and ergonomics for ITS, the objective of the joint 
research activities is to converge, in the coming decade, towards a coherent 
research programme with complementary approaches. 
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Fig. 1. HUMANIST task forces 
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It is not intended in this paper to make an exhaustive presentation of the 7 
research Task Forces, but rather to give examples about the topics that have 
been more specifically developed during the past year. 



3.1 Task Force A: Identification of the Driver Needs in Relation to ITS 

Identification of drivers needs according to new technological development 
requires several types of investigations and a lot of effort, as there is a wide 
heterogeneity of the population in terms of functional abilities and require- 
ments. 

The methodology to investigate driver needs covers a wide area. Driver needs, 
especially for driving task support, are investigated often by analysing driver 
errors and accidents. Driver and traveller information needs are often studied 
by interviews and questionnaires. 

Age and driving experience seem to be the most important factors when defin- 
ing driver needs for support. Especially, the elderly would need support in 
many situations, e.g. merging and turning in intersections. There are, howev- 
er, not many systems available for supporting them, yet. Another group with 
high relative risk in traffic is young drivers, especially male. 

One specific group to take into account is professional drivers. They need com- 
munication systems while driving, more often than private drivers. 
Communication while driving is often highly important for their work, and 
therefore, this fact must be taken into account while planning different restric- 
tions for the use of communication aids. Use of alcohol and/or drugs is an 
important factor in traffic safety work. 



3.2 Task Force C: Joint Cognitive Models of the Driver-Vehicle-Environment for 
User centred Design 

In order to produce really suitable and customised design solutions for the 
automotive domain, in-vehicle systems should be designed following the 
approach known as User-Centred Design. A crucial element in the UCD 
process is the overall model of the "joint" cognitive system represented by the 
Driver, Vehicle and Environment (DVE Model). In the framework of HUMAN- 
IST activities, a book about Modelling Driver Behaviour in Automotive 
Environments has been published; it gathers the state of the art in modelling 
and simulation of DVE systems and in particular of driver behaviour that can 
be applied and used in nowadays and tomorrow automotives. 
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3.3 Task Force D: Impact Analysis of ITS on Driving Behaviour 

With the increasing development and implementation of ITS on the market, 
the issue of assessing and evaluating their effects on various road traffic relat- 
ed criteria becomes more and more up to date. Thus, one main objectives of 
this Task Force is to describe and continuously update the state-of the-art on 
the impact of in-vehicle information and communication systems (I VIS) on rel- 
evant criteria of driver workload and behaviour. 



3.4 Task Force E: Development of innovative Methodologies to evaluate IT 
Safety and Usability 

Knowledge and experience of partners, about methodologies for the evaluation 
of IVIS both in terms of safety and usability, have been shared in order to build 
up a Matrix of Methodologies. Objective of this matrix was to identify and to 
categorise existing and proposed methods for evaluation of ITS. When assess- 
ing usability and safety, different metrics are used. These metrics are collect- 
ed using specific techniques and often require specific tools. The metrics are 
collected in one or more physical environments. The combination of 
metric/technique/ tool/ environment is what is referred to as a method. 
Additional information in the matrix includes the type of data obtained; the 
effectiveness of the method, practical issues of application, scientific refer- 
ences and a list of HUMANIST partners with experience or knowledge of the 
methods. Partners were asked to indicate their favourite methods. From this, a 
selection of 22 broad ranging methods were indicated as 'favourites', demon- 
strating the variety of methods available and that methods used depend heav- 
ily on the objectives of a study. The matrix can be updated as new information 
becomes available and can serve as a source of reference. It will form the basis 
for consideration of more integrated methodologies within the project. 



3.5 Task Force G: Use of ITS to train and to educate Drivers 

Driving simulators and e-learning communication technologies hold consider- 
able promise for enhancing driver training, testing, and licensing and improv- 
ing road safety. One of this Task Force objectives is to assess the present state 
of the art in the application of simulators to driver training, and to identify 
what additional R&D effort is needed to achieve the potential effectiveness in 
using simulators to improve the driver training process at a European level 
with harmonized procedures. At this stage, some main statements can be 
made: 
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► Driving simulators or e-learning are already being used in driver train- 
ing, both of novice drivers and of professional drivers. On the contrary 
there are few alternatives to cover the needs of disabled or elderly driv- 
ers. 

► The validation of simulators is also an area where there is still a need 
for further advances. A methodological approach to driver simulator 
validation for training is required. 

► In parallel, teaching skills have to improve, by including these new 
methodologies in a global training curriculum, by applying them to the 
different driving task levels of matrix and by developing performance 
measurement tools to monitor in real-time driver's progress and to 
improve the trainee's feedback. 




Fig. 2. Four generations of driver train- 
ing simulators; TNO Human 
Factors, The Netherlands 



Fig. 3. Simulated urban traffic envi- 
ronment for a bus drivers train- 
ing simulator developed at 
Madrid Polytecnic University 



4 Integration 

The set of integrating activities has been created in order to manage and to 
consolidate the NoE research structure. 

In order to promote the mobility of researchers inside Europe, a number of 
PhD thesis have been funded by HUMANIST in the following topics: 

► Development of assessment methodologies 

Definition of reference interfaces and reference models of driver inter- 
action for comparing the distraction potentials of in-vehicle devices 
Feasibility study using Event-Related Potentials to analyse the effects of 
secondary driver assistance information on primary driving task infor- 
mation extraction and processing 
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► Study of the impact of ITS enforcement systems on driver behaviour 
Influence of speed adaptation system on the communications between 
road users in urban areas. Investigation of the psychological, legal and 
technical issues of video enforcement systems to reduce driving 
offences and to quiet drivers' behaviour 

► Study of the influence of ITS on driver behaviour 

Investigation of which components of the situation awareness could be 
affected by driving with ITS Study of the impact of multiple visual and 
auditory inputs on driver behaviour, by identifying the age-related dif- 
ference in the driving task performance. Subjective evaluation by pro- 
fessional drivers of the impact of on-board devices, on driver's stress 
and mental workload 

► Driver modelling and cognitive simulation 

Driver behaviour modelling and cognitive engineering tools develop- 
ment in order to assess driver's situation awareness 
Analysis and modelling of driving activity and driver's behaviour for 
the design of on-board systems 
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Fig. 4. HUMANIST activities 

The following Post-Doc topics will be also supported by the HUMANIST NoE 

► Information technology and older female drivers 

► Incorporation of parameters related to its Into traffic simulation models 

► The effects of in-car speed information systems on Driving behaviour 

► Situation awareness when using in-vehicle Systems 

► Its and unprotected road users - challenges and potential 

► Nomadic devices - use and impact on driving behaviour 
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5 Dissemination 

There is a set of activities devoted to organise and to strength exchanges 
between research activities developed inside the network, and relevant stake- 
holders belonging to various areas of interest: automotive industries, transport 
services, European Integrated Project in this area, EC representatives from the 
eSafety initiative. Standardisation bodies, European Universities, Final users. 
Insurance companies. 




Fig. 1 . HUMANIST dissemination 

This activity is developed through the organisation of Stakeholders Forum, 
with interactive debates between participants from various backgrounds and 
the HUMANIST partners. An electronic forum available for open discussion 
will be set up on the Electronic Collaborative Work Environment with a dedi- 
cated access to identified users. This access will allow external stakeholders to 
be able to raise specific issues, and to communicate with HUMANIST partners 
in an interactive way in order to get feedback regarding some of their con- 
cerns. 

A training program has been set up toward European students, young 
researchers and professionals of the field willing to be informed about new 
development of the research in the area of ITS design. The training program 
for professionals is open to any interested stakeholders and is composed by lec- 
turers and by practical exercises. 
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High Speed 1 Gbit/s Video Transmission with Fiber Optic 
Technology 

T. Wipiejewski, F. Ho, W. Hung, S. Cheng, E. Wong, St. Ng, W. Ma, Th. Choi, 
G. Egnisaban, K. Yau, T. Mangente, ASTRI 



Abstract 

We demonstrate a 1 Gbit/s video transmission over up to 15 m of 
polymer cladded silica (PCS) optical fiber using high speed fiber 
optic transceiver (FOT) technology. The video link consists of a cam- 
era with an electrical S800 LVDS interface and a high resolution 
monitor. A pair of fiber optic cable connects the two ports replacing 
and extending the regular electrical cable. In contrast to the electri- 
cal cable the optical fiber is thin, light weight, and highly flexible. 

The high power budget margin of the link allows 3 in-line connec- 
tors for the 10-15 m link length. For other applications a longer link 
length is feasible using higher bandwidth optical fiber. 

The FOTs are high speed versions based on the technology we have developed 
for the 50 Mbit/s infotainment network. The transmitter contains a vertical- 
cavity laser (VCSEL) light source operating in the infrared wavelength regime 
at 850 nm, an electronics driver chip, and some passive components. The pack- 
age is a low cost leadframe style for through-hole or surface mounting. The 
output power of the transceiver is limited to 0.7 mW to meet eye-safety class 
I requirements. An internal control circuit stabilizes the optical output power 
over the entire operating temperature range from -40°C to + 105°C to within 1 
dB. Due to an integrated lens we achieve a wide coupling tolerance of the 
VCSEL into the PCS fiber of 480 mm over a long longitudinal tolerance range 
of 0.5 mm. The large alignment tolerance ensures a stable coupling over tem- 
perature and time and enables the usage of very low cost plastic molded con- 
nector ferrules. 

The FOT receiver module contains a high speed MSM photodetector. The spe- 
cial design of the MSM photodetector enables high speed operation up to 3.2 
Gbit/s in combination with a large active area for easy alignment. 
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1 Introduction 

Photonics technology has been applied in many ways for automobiles. As 
shown in Fig.l photonics applications in automobiles can be categorized into 
three main areas which are connectivity lighting, and sensing. Light emitting 
diodes (LED) have been used for dash board illuminations for many years, 
because of its color purity small size, and high efficiency. White light LEDs are 
now making it into interior lighting. LEDs are used for brake lights and some- 
times for turn signals, due to their long life time and high efficiency. A new 
stage of development is the replacement of light bulbs for front lights such as 
fog lamps. Optical sensors have also been used in a variety of applications for 
measuring different physical or chemical parameters. New developments 
include examples such as optical fiber sensors for seat occupancy detection, 
light curtains for window blocking detection, infra-red sensors for night vision, 
and others [1-4]. Fiber optic connectivity has been implemented for the info- 
tainment system of many automobiles. MOST and IDB1394 are standards for 
the it [5-6]. The typical data rate is 50 Mbit/s for MOST and 250 Mbit/s for IDB 
1394 S200. These numbers are the physical data rates including some over- 
head load due to the implemented coding schemes. 

Many European car manufacturers have already adopted the MOST system in 
their car models. The fiber optic transceiver production volume for automotive 
applications is already more than ten million units per annum. Besides the 
infotainment system fiber optics has been introduced to the safety system. 
The Byteflight system of BMW [7] is pioneering fiber optic links in their airbag 
control system. Today, camera and video links are under development for driv- 
er assistance to provide visual information in the applications such as lane 
detection, traffic sign recognition, or object identification. These applications 
require even higher data rates of 500 Mbit/s up to 1 Gbit/s (IDB 1394 S800) to 
transmit uncompressed visual data for real time reactions. 
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Fig. 1 . Photonics applications in automobiles 
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At high data rates electrical cables require relatively good shielding to mini- 
mize electromagnetic interference (EMI). Both coaxial type cables and shield- 
ed-twisted-pair (STP) become quite bulky It is also more difficult to include in- 
line connectors, because of the shielding and required mechanical precision. To 
overcome these drawbacks, we have developed a 1 Gbit/s video transmission 
over up to 1 5 m of polymer cladded silica (PCS) optical fiber using high speed 
fiber optic transceiver (FOT) technology. The video link consists of a camera 
with an electrical S800 LVDS interface and a high resolution monitor. A pair of 
fiber optic cables connects the two ports replacing and extending the regular 
electrical cable. In contrast to the electrical cable the optical fiber is thin, light 
weight, and highly flexible. The high power budget margin of the link allows 
3 in-line connectors for the 10-15 m link length. For other applications a longer 
link length is feasible using higher bandwidth optical fiber. 

The infotainment system in automotive usually includes a variety of devices 
like digital radio, DVD/CD player, video displays or GPS navigation system to 
provide on-road entertainment and information to the drivers and passengers. 
In many European cars which adopting MOST standard, these devices are con- 
nected with optical links as shown in Fig. 2. A basic optical link consists of a 
pair of optical cables and two pairs of fiber optic transceivers which are 
installed in the device. The fiber optic transmitter converts the electrical sig- 
nal form electronics board into optical signals and transmits them through the 
optical cable. The fiber optic receiver detects the optical signals from the opti- 
cal cable and coverts them back into electrical signals for the electronics board 
circuitry. The transmission distance of the optical link is usually from 10 cm to 
30 m. The key benefit of fiber optic link is the absence of electromagnetic 
interference (EMI) noise creation and susceptibility. In addition, fiber optic 
cables are thin, light weight, and highly flexible. The fiber optic connectors are 
robust and can carry high speed signals up to Gbit/s easily. 




Fig. 2. A basic optical link in automotive infotainment system 
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Currently plastic optical fiber (POF) and 650 nm LEDs are used in the MOST 
fiber optics system [1]. This system run with a power budget of around 14 dB 
over an operating temperature range from -40 °C to +85°C or +95°C. 
Therefore the current fiber systems are limited to the passenger compart- 
ment. For roof top implementations a maximum temperature of 105°C would 
be required and for the engine compartment even up to 125°C. PCS fiber is 
under consideration to overcome the current operating temperature limita- 
tions. This fiber can be operated even at 125°C. Tab. 1 summarizes the techni- 
cal requirements of different standards and corresponding fiber optics tech- 
nologies. The IDB1394 standard has four standard data rates ranging from 100 
Mbps to 800 Mbps classified as S100, S200, S400 and S800. POF and 650 nm 
LED are proposed for the SI 00, if the operating temperature up to 85°C is suf- 
ficient. For S200, some people use 650 nm resonant cavity (RC) LED as light 
source to work with PMMA POF [8-9], however, the operating temperature is 
still limited up to 85°C or 95°C. For higher temperature (105oC) and higher 
data rates (S400 and S800) PCS fiber and 850 nm VCSEL provide the desired 
performance and relatively low cost solution. 
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Tab. 1. Summary of the technical requirements of different communica- 
tion standards and corresponding fiber optics technologies 



2 Fiber Optic Transceivers 

2.1 Through-Hole Package Design 

Based on conventional semiconductor packaging technologies, we have devel- 
oped VCSEL based ultra small form factor fiber optic transceiver (FOT) mod- 
ules for large core fiber systems. A photo of the FOT modules is shown in Fig. 
3. The outer dimensions of 9.7 mm x 6.2 mm x 3.6 mm are very small com- 
pared to standard SFF (small form factor) modules used in data communication 
for computer networks. The opto-electronic components and ICs are directly 
assembled on a lead frame platform and encapsulated with polymers with 
suitable mechanical and optical properties. The maximum operating tempera- 
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ture is up to +105°C, much higher compared to typical datacom ( + 85°C) 
devices. 

The transmitter and receiver modules are mirror images of each other. They 
both contain four electrical leads for through-hole mounting. Two are for 
power supply one for data channel, and one for control purposes. There is a 
wake-up and sleep mode function implemented in the design for power saving 
applications. 



Fig. 3. High speed fiber optic transmitter for a wide operating tempera- 
ture from -40 °C to + 105°C. 



2.2 Output Power Stability of Transmitter Module 

One critical design concern of the VCSEL transmitter module is to stabilize the 
optical power coupled from VCSEL to the optical cable over wide temperature 
range. Our approach is to use a monitoring photo detector (MPD) to receive 
part of the optical output and feedback the photocurrent into the laser diode 
driver. The laser diode driver controls the bias current to the VCSEL automat- 
ically to stabilize the optical output power. Besides, a barrel feature is inte- 
grated with a lens in the module housing to facilitate an easy and robust cou- 
pling to the external optical cable. Fig. 4 shows the schematic diagram of the 
transmitter module with the optical feedback loop and coupling optics. 



Vcc 




Fig. 4. Schematic diagram of the VCSEL transmitter module. 
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From the simulation results as shown in Fig. 5, the optical coupling design of 
transmitter provides loose alignment tolerances of 480 mm in lateral axis for 
a working distance of 500 mm to a 200 mm core diameter PCS optical fiber. 
The coupling optics for the receiver is optimized for focusing the light from the 
fiber facet to the large area photodetector. The alignment tolerances are simi- 
lar to the transmitter. We have confirmed our simulation results with experi- 
mental data as well. The wide alignment tolerance enables our customers to 
use very low cost plastic molded connectors with relaxed tolerance for their 
mechanical dimensions. 




Fig. 5. Simulated coupling tolerances of VCSEL FOT to a 200 mm core 
diameter PCS fiber. 



Fig. 6 shows the coupled power of the VCSEL transmitter into a 200 mm core 
diameter PCS fiber cable over a wide temperature range from -40 °C to + 105°C. 
A variation of less than is measured. This excellent value enables a tight out- 
put power specification of the transmitter. The tight power specification 
results in a better link margin for the transmission system. 
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Fig. 6. Coupling power of the VCSEL FOT to a 200 mm core diameter PCS 
fiber over temperature from -40°C to + 105°C. 
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2.3 Surface Mount Package Design 

Surface-mount-technology (SMT) has been widely used in electronics board 
assembling. Compared to the through-hole mounted device surface mount 
devices (SMD) have a number of advantages such as smaller size, higher 
degree of assembling automation, and higher circuit density Components can 
be mounted on both sides of the board. The are more robust under mechani- 
cal stress and have a lower lead resistance and inductance leading to better 
performance for high frequencies. Adopting SMT in opto-electronic packaging 
is very attractive not only due to the advantages of this technology but also 
because the infrastructure of electronics manufacturing can be leveraged. 
However, it has been very difficult to make optical transceiver modules that 
are compatible with the reflow soldering process of SMT devices, because of 
the material used to the optical components such as lenses and optical cou- 
pling window. 

We develop surface mount type packages for our fiber optic transceivers. This 
is to complement and extend our through-hole package product line. Fig. 7 
shows a photograph of an SMT FOT prototype. It has a similar form factor as 
the through-hole type fiber optic transceivers. One challenge is to select the 
right molding material which has the suitable optical properties for efficient 
optical coupling and thermal properties compatible to reflow soldering. 




Fig. 7. Photograph of an SMT fiber optic transmitter module prototype. 

We have verified the reflow soldering compatibility of the SMT fiber optic 
transmitter module by measuring the optical alignment performance before 
and after undergoing a reflow soldering process. The reflow soldering tem- 
perature profile is set to 240°C for 46 seconds and to peak at 250°C. This pro- 
file complies with Pb-free soldering requirements. Fig. 8 depicts the optical 
power coupled from 5 samples of SMT fiber optic transmitter modules before 
and after undergoing the reflow soldering process, plus one control sample 
which has not undergone reflow soldering. No significant change in the cou- 
pled power is observed indicating the good stability of the packages. 
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Fig. 8. Optical power coupled from five samples of SMT fiber optic trans- 
mitter modules before and after undergoing the reflow soldering 
process with 240 °C for 46 seconds and a peak temperature of 
250°C for lead-free soldering. 



3 Transmission Performance 

3.1 Transmission Performance over Wide Temperature Range 

The current MOST infotainment system runs at a data rate of 50 Mbit/s. The 
VCSEL based FOT can easily meet the timing specifications of the link. The fast 
turn-on and turn-off signal transitions provide a low jitter value for the entire 
optical system. Fig. 9 depicts the eye diagrams of the transmitter module at 
50 Mbit/s data rate for different temperatures of -40°C, +25°C, and +105°C. 
The eyes diagrams are wide open indicating a good and error free transmis- 
sion performance. The extinction ratio is greater than 10 dB for all tempera- 
tures. The uncorrelated jitter is less than 150 ps. 

Fig. 10 shows the eye diagrams of the Si-pin PD receiver module at 50 Mbit/s 
data rate for different temperatures of -40°C, +25°C, and +105°C. The eyes 
diagrams are wide open showing an excellent, error free transmission per- 
formance. The optical modulation amplitude of the input signal is -24 dBm. 
The uncorrelated jitter is less than 800 ps. 
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(a) (b) (c) 

Fig. 9. Eye-diagrams of the transmitter module at 50 Mbit/s data rate at 
an ambient temperature of (a) -40 °C, (b) + 25°C, and (c) + 105°C. 




Fig. 10. Eye-diagrams of the receiver module at 50 Mbit/s data rate at an 
ambient temperature of (a) -40 °C, (b) +25°C, and (c) +105°C. 



3.2 High Speed Transmission Performance 

For higher speed applications such as the IDB 1394b standard and video links, 
we have demonstrated the high speed performance of a pair of VCSEL trans- 
mitter and GaAs metal-semiconductor-metal (MSM) photodetector receiver 
over 10 m of PCS fiber as shown in Fig. 11. 




Fig. 11. Experimental setup of transmission over PCS fiber with VCSEL 
transmitter and MSM-PD receiver 



The VCSEL transmitter is capable of much higher data rates from a few hun- 
dreds Mbit/s to Gbit/s range with a different electrical circuit design. The spe- 
cial design of the MSM photodetector enables receiver module to operate at 
3.2 Gbit/s in combination with a large active area for easy alignment. Figure 
12 depicts eye diagrams of the transmission at 300 Mbit/s, 600 Mbit/s, and 
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1.25 Gbit/s, respectively. The extinction ratio is greater than 9 dB in all cases. 
The total jitter is around 100 ps. 




(a) (b) (c) 

Fig. 12. Eye-diagrams after 10 m of PCS fiber transmission at (a) 300 
Mbit/s, (b) 600 Mbit/s and (c) 1.25 Gbit/s. 



4 Optical Video Link 

We have set up an IEEE 1394b S800 video link with the fiber optic transceivers 
and PCS fiber. As shown in Fig. 13, the video link consists of a camera with an 
electrical S800 LVDS interface and a high resolution monitor. A pair of fiber 
optic cable connects the two ports replacing and extending the regular elec- 
trical cable. In contrast to the electrical cable the optical fiber is thin, light 
weight, and highly flexible. 




Adapter 

board 



Fig. 13. System architecture of the IEEE 1394b S800 optical video link. 



We have measured the bit error ratio (BER) of the optical links with fiber 
length of 1 m, 5 m, 10 m, and 15 m as shown in Fig. 14. 

In this measurement, a 2 7 -1 pseudo-random bit sequence (PRBS) data stream 
at 1 GHz clock frequency is used. The launch power of fiber optic transmitter 
is -1.5 dBm. The power budget for the four different optical links are inferred 
from the BER measurement and listed in Tab. 2. We assume a 2 dB power 
budget margin and a transceiver coupling loss of 2 dB. The insertion loss for 
each connector is also 2 dB. We calculate the maximum numbers of in-line con- 
nectors for each optical link as listed in Tab. 2. The power budget margin of 
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the link allows 3 in-line connectors for the 10-15 m link length. For other appli- 
cations a longer link length is feasible using higher bandwidth optical fiber. 



* 1m ■ 5m a 10m 15m 




Fig. 14. Bit error ratios of the optical links with fiber length of 1 m, 5 m, 10 
m and 15 m. A 27 -1PRBS data stream at 1 GHz clock frequency is 
used. 



Transmission distance 


lm 


5m 


10m 


15m 


Error-free operating power (dBm) 


- 15.5 


-15 


- 13.0 


- 12.0 


Maximum power budget (dB) 


14 


13.5 


11.5 


10.5 


Approx, number of in-line connectors allowed 


5 


4 


3 


3 



Tab. 2. Power budget and maximum number of in-line connectors for the 
optical video link of different length. 



5 Conclusion 

We have demonstrated a 1 Gbit/s video transmission over up to 1 5 m of PCS 
optical fiber using high speed FOT technology. The high power budget margin 
of the link allows 3 in-line connectors for the 10-15 m link length. For other 
applications a longer link length is feasible using higher bandwidth optical 
fiber. The FOTs are high speed versions based on the technology we have 
developed for the 50 Mbit/s infotainment network. The FOT package is based 
on low cost leadframe and plastic molding technologies. Packaging formats for 
through-hole or surface mounting are designed. The transmitter contains a 
VCSEL light source operating in the infrared wavelength regime at 850 nm, an 
electronics driver chip, and some passive components. The output power of 
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the transceiver is limited to 0.7 mW to meet eye-safety class I requirements. 
An internal control circuit stabilizes the optical output power over the entire 
operating temperature range from -40 °C to + 105°C to within 1 dB. Due to an 
integrated lens we achieve a wide coupling tolerance of the VCSEL into the 
PCS fiber of +B0 mm over a long longitudinal tolerance range of 0.5 mm. The 
large alignment tolerance ensures a stable coupling over temperature and 
time and enables the usage of very low cost plastic molded connector ferrules. 
The FOT receiver module contains a high speed MSM photodetector. The spe- 
cial design of the MSM photodetector enables high speed operation up to 3.2 
Gbit/s in combination with a large active area for easy alignment. The appli- 
cations include video networks, information displays, and high speed cameras. 



Acknowledgement 

We gratefully acknowledge great support from our suppliers and partners. 
This work was financially supported by funds from the Hong Kong Innovation 
and Technology Commission ITC. 



References 

[1] H. Poisel, M. Luber, O. Ziemann, "POF sensors for Automotive and Industrial Use 
Come of Age," Proc. 14th Inti. Conf. Polymer Optical Fiber, pp. 285-289, Sep, 2005. 

[2] G. Kodl, G. Richinger, Ch. Weiss, " Dirtiness Sensor, based on the Changes in the 
Evanescent Field," Proc. 14th Inti. Conf. Polymer Optical Fiber, pp. 2 2 5-2 2 8, Sep, 
2005. 

[3] H. Y. Liu, H. B. Liu, G. D. Peng, P. L. Chu, " Strain Sensing using a Fiber Laser and 
a Polymer Optical Fiber Bragg Grating," Proc. 14th Inti. Conf. Polymer Optical 
Fiber, pp. 229-232, Sep, 2005. 

[4] B. L. Henoret, M. Miedreich, "Fiberoptic sensor for pedestrian protection", Proc. 
13th Inti. Conf. Polymer Optical Fiber, pp386-392, Sep, 2004. 

[5] Eberhard Zeeb, "MOST-Physical Layer Standardization Progress Report and Future 
Physical Layer Activities", Proc. The 3rd Automotive LAN Seminar, pp. (3-3) - (3- 
34), Oct, 2003. 

[6] W. Daum, W. Czepluch, "Reliability of Step-Index and Multi-Core POF for 
Automotive Applications," Proc. 12th Inti. Conf. Polymer Optical Fiber, pp.6-9, 
Sep, 2003. 

[7] R. Griessbach, J. Berwanger, M. Peller, "Byteflight - neues Hochleistungs- 
Datenbussystem fur sicherheitsrelevante Anwendungen," ATZ/MTZ Automotive 
Electronics, Friedrich Vieweg & Sohn Verlagsgesellschaft mbH, January 2000. 



High Speed 1 Gbit/s Video Transmission with Fiber Optic Technology I 319 



[8] J. D. Lambkin, T. McCormack, T. Calvert, T. Moriarty, B. McGarvey, d. O'Mahoney, 
" 650nm RCLEDs and VCSELs for IEEE 1394b Applications" Proc. 12th Inti. Conf. 
Polymer Optical Fiber, post-dead line paper, Sep, 2003. 

[9] John D. Lambkin, "RCLEDs for MOST and IDB 1394 automotive Applications", 
Proc. 14th Inti. Conf. Polymer Optical Fiber, pp.51, Sep, 2005. 

Torsten Wipiejewski 

ASTRI, 5/F, Photonics Center 
Hong Kong Science Park 
Hong Kong 
torsten@astri.org 

Keywords: video transmission, fiber optic technology, fiber optic transceiver, VCSEL 

FOT, high speed photodetector, MSM photodetector, PCS optical fiber 
link, infotainment system 



323 



Integrated Giant Magneto Resistors - a new Sensor 
Technology for Automotive Applications 

W. Rossler, J. Zimmer, Th. Bever, K. Priigl, Infineon Technologies AG 
W. Granig, D. Hammerschmidt, E. Katzmaier, Infineon Technologies Austria 
AG 

Abstract 

The paper will give an introduction to the principle of the giant mag- 
neto resistive - GMR - effect and the silicon system integration of 
GMR sensors. The two main applications of a GMR are as a magnet- 
ic field strength sensor and as an angular field direction sensor. They 
will be discussed under consideration of automotive requirements. 

The typical applications of a magnetic field strength GMR sensor in 
incremental position and speed sensing and those of GMR angular 
field sensors in position sensing will be summarized. Finally advan- 
tages of GMR in those applications will be discussed and conclusions 
on the use of GMR in automotive sensing will be drawn. 



1 Introduction 

The giant magneto resistive effect, which was discovered 1988, is the result of 
magnetic coupling in a stack of ferromagnetic layers. They are separated by 
non-ferromagnetic spacer layers. This effect can be used to construct different 
types of magneto resistive sensors. For our applications we focus on so called 
'spin valve structures', which allow covering field strength sensitive or field 
direction sensitive constellations using the same basic technology. 

For large scale introduction of GMR sensors in automotive applications a mass 
fabrication technology for the sensors is required. The most advanced high 
volume fabrication technology nowadays is CMOS (Complementary Metal 
Oxide Semiconductor) integrated circuit processing. CMOS in general is a good 
candidate for GMR integration. The techniques that are used in CMOS process- 
es include methods for sputtering and structuring of thin layers of different 
materials. Alternatively bipolar technologies can be used for the basic signal 
processing chip. 
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Furthermore there is experience in handling magneto resistive structures 
available from the MRAM (Magneto Random Access Memory) technology 
which is valuable for process optimization and fast ramp up of the new sensor 
technology. 

The monolithic integration of the GMR will be focused on in the fourth section 
of this paper. The use of magnetic field sensors in modern cars is multifarious. 
The state-of-the-art technology uses the Hall effect. Besides simple magnetic 
switches there are two different basic modes of operation for automotive Hall 
sensors. These are incremental position and speed detection as the first and 
linear position and angle measurement as the second group of applications. 

If a new technology is introduced there should be a significant advantage over 
the existing one. Thus, we will discuss the incremental, linear and angle appli- 
cations more in detail in the fifth and sixth section of this paper and compare 
the existing Hall solutions with the potential that the GMR technology may 
offer [1]. 



2 The GMR Effect 



The giant magneto resistive (GMR) effect can be explained as interface scat- 
tering when the mean free paths of minority spin and majority spin carriers 
of conducting electrons through successive magnetic layers alters. The mag- 
netic layers are separated by a spacer layer, which is either a non-ferrous 
metal or a noble metal. Ferromagnetic transition metals like Ni, Co or Fe are 
commonly used as magnetic layers. The metal layers should be very thin such 
that the mean free path of the spin of the conducting electrons can be 
assumed to be much longer. 



4. FM 4 






Fig. 1 . Density of states model for a basic GMR system 
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The carriers can either exhibit spin-up or spin-down characteristics. Fig. 1 
shows schematically the density of states of the s- and d-band electrons as a 
function of energy for anti-ferromagnetically coupled two layer system. The 
minority carriers exhibit a higher resistivity compared to the majority carriers 
due to a higher scattering probability of the d-band electrons (lower mobility). 
In the case when the magnetic layers have antiparallel aligned magnetization 
directions majority carriers traveling from one FM layer to the other FM layer 
become minority carriers combined with a higher resistivity there. Thus, anti- 
parallel alignment of neighboring FM layers results in an enhanced resistance. 

In the case of parallel alignment majority carriers traveling from one to the 
other FM layer are still majority carriers there, resulting not in an enhanced 
resistance as is demonstrated in Fig. 2. This observation allows using the Mott 
two-current model. It assumes that the total resistance can be expressed as the 
sum of the separate contributions from spin-up and spin-down electrons, 
which can be seen as two separate channels added in parallel (Fig. 3) 





parallel magnetization anti-parallel magnetization 

Fig. 2. Scattering of spin up and spin down carriers in a multilayer GMR 
structure 




Fig. 3. Mott two current equivalent circuit 



The GMR effect can be enhanced by increasing the delta resistance between 
spin-up and spin-down channels. 
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(i) 



(2) 



GMR rel 




(3) 



However, spin mixing can occur which lowers the GMR effect. For optimiza- 
tion of the GMR effect, micro structural features like grain size and texture 
and circuit configurations like Current-In-Plane (CIP) or Current-Perpendicular- 
to-Plane (CPP) have to be considered. 



3 GMR Spin Valve System for Sensor Application 

3.1. GMR Spin Valve 

GMR multilayer configurations were the first structures suitable for device 
application. They consist of typically up to 20 non-ferromagnetic/ferromag- 
netic bi-layers resulting in high GMR values. They feature a bell-shaped resist- 
ance change with magnetic field and are suitable to detect in-plane magnetic 
field strengths independently of the field direction. However they can exhibit 
a pronounced hysteresis which complicates the application of this sensor type 
for highly accurate automotive magnetic field sensors. Furthermore, the bell- 
shaped signal characteristic of the multilayer structure can lead e.g. in the 
presence of an interfering magnetic field to a frequency doubling when meas- 
uring an oscillating magnetic field, since there is no clamping of the signal for 
extreme magnetic field values. 

GMR spin valve structures possess improved properties combining high GMR 
values with low hysteresis, signal clamping and high stability. They were 
developed in the early 1990ies for read heads in magnetic storage devices. 
Compared to the GMR multi layer structure, the magnetic properties of a spin 
valve stack can be adapted to the specific application so that the measurement 
of the strength and/or the direction of a magnetic field are possible. 

As described above, GMR spin valve systems basically consist of two ferro- 
magnetic layers separated by a non-magnetic spacer layer as depicted in Fig. 
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5. One layer is the 'hard' or 'pinned' layer and the other the 'soft' or 'free' 
layer. In contrast to the mulilayer system the pinned layer's magnetization 
direction is ideally fixed whereas the free layer magnetization can follow an 
external in-plane magnetic field with almost no hysteresis. 

A defined alignment of the magnetization directions in the hard layer is 
achieved by "pinning" to a synthetic anti-ferromagnet (SAF) system. The SAF 
benefits the high stability of the system, because the pinned hard layer pos- 
sesses a large magnitude of unidirectional anisotropy that is the tendency of a 
ferromagnet to have a preferential direction of magnetization. The hard layer 
is magnetically biased in direction of the 'easy axis' by the means of the 
exchange bias effect. The exchange bias describes the pinning of the magne- 
tization of a ferromagnetic layer by an adjacent anti-ferromagnet. 

The second magnetic layer, the so-called free layer is separated by a nonmag- 
netic spacer with an adjusted thickness resulting in an ideally zero coupling 
between the magnetic layers. Coupling between 'soft' and 'hard' layer is very 
weak, ideally zero. However, coupling can occur from residual exchange bias 
fields, interface roughness or magnetostatic coupling to neighbouring ferro- 
magnetic layers. 




Fig. 4. Simplified spin- valve GMR stack 

More sophisticated spin-valve systems have implemented a tri-layer artificial 
anti-ferromagnet instead of the pinned layer. This so-called reference system 
consists of a pinned layer which is exchange coupled to a natural anti-ferro- 
magnet and an anti-ferromagnetically coupled reference layer. The (valuable) 
GMR effect takes place between the free and the reference layer. The advan- 
tage of such a reference system is the much lower net magnetic moment com- 
pared to the case with a pinned layer only which stabilizes the reference mag- 
netization in an external magnetic field. 
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Fig. 5. Minor loop of a GMR magnetization curve 

Fig. 5 shows a typical easy-axis (i.e. magnetic field parallel to the reference 
magnetization axis) minor loop characteristic of a GMR spin valve magnetiza- 
tion curve reflecting basically the free layer properties. H e denotes the resid- 
ual ferro-/antiferromagnetic coupling and H c is the coercivity of the free layer. 
The dependence of the GMR value as a function of the magnetic field leads to 
the possible application as a magnetic switch where high sensitivities (small 
switching fields) and small coercivities are needed.. 




Fig. 6. Major loop (±200 mT) of a GMR magnetization curve 

In Fig. 6 presents the corresponding magnetic easy-axis behavior in the range 
+ 200 mT which characterizes mainly the properties of the spin-valve refer- 
ence system. The stable plateaus in the low/high resistance regime beyond 
120 mT demonstrates that the spin-valve structure can be designed to feature 
a stable reference system for magnetic field strengths which are relevant for 
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automotive applications. Furthermore, the exposure of even higher fields does 
not lead to irreversible changes of the sensor properties. 



3.2 GMR Spin Valve System for Sensing the Angle of Magnetic Field 

For magnetic fields in the range of some 10 mT, i.e. in the saturated range, the 
free layer is aligned to the direction of the external field. In this case the resist- 
ance of the GMR stack is a function of the cosinus of the angle between the 
magnetizations of free and pinned layer (Fig. 4). While the magnetization in 
the free layer follows the external magnetic field, the magnetization in the 
pinned layer is fixed by the exchange coupling in a reference direction. This 
reference direction is defined during the magnetization process using high 
temperatures and high magnetic fields. 




Fig. 7. Resistance of a GMR spin valve operated in saturation as a function 
of the field angle with respect to the magnetization direction of the 
pinned layer. 

As shown in Fig. 7, the cosine dependence of the GMR effect upon rotating the 
free layer magnetization direction under a constant external field in the satu- 
ration region leads to the possible application as a rotary angle sensor. The fol- 
lowing equation describes the relationship. 



<*„-*,) -CO S<A0) (4) 



whereas AQ denotes the angle between the magnetization direction of the 
pinned and free layer and R 0 the resistance in the absence of any magnetore- 
sistive effects. fi p is the resistance when magnetization directions between 
pinned and free layer are in parallel and ft ap is the resistance when the mag- 
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netization directions between pinned and free layer are anti-parallel. As a 
result, a single GMR resistor exhibits a 180° uniqueness. Compared to the 
AMR effect which has a angle uniqueness in 90° the GMR spin-valve structure 
features a significant benefit. As a result, a 360° angle sensor is enabled by 
using a bridge circuit having GMR resistances with reference directions ori- 
ented along two axis only. 



3.3 GMR Spin Valve System for Sensing the Strength of Magnetic Field 

For incremental speed sensors an approximately linear MR behavior around 
zero field is favorable in order to enable tooth decoding (correct identification 
of tooth characteristic) of a tooth wheel e.g. for a "true power on" function. If 
the magnetization direction of the reference layer is aligned perpendicular to 
the magnetization / anisotropy axis of the free layer, a linear MR behavior with 
respect to a magnetic field component parallel to the reference magnetization 
is obtained. The magnetic field which is necessary for a fully anti-parallel (-H k ) 
and parallel ( + H k ) alignment of the free and reference layer describes the 
maximum linear range. Since the H k is a property of magnetic anisotropy of 
the free layer only, sophisticated material and/or geometry engineering can be 
applied to adapt the magnetic performance. By choosing special growth con- 
ditions for the free layer it is possible to induce a certain anisotropy axis which 
can lead to an H k greater than 8 mT. Furthermore, there are ferromagnetic 
materials where an anisotropic axis can be induced by an anneal in a magnet- 
ic field, thereby obtaining an H k as high as 5 mT. Another possibility to tailor 
a certain strength as well as direction of the magnetic anisotropy axis is to 
make use of the shape anisotropy by adjusting the geometry of the GMR 
resistance. In order to minimize the free magnetostatic energy it is favorable 
for the magnetic layers to align the magnetization along the long axis e.g. of a 
resistance stripe. The higher the aspect ratio of the GMR structure is, the 
stronger is the magnetostatic energy. 

Fig. 8 shows the simulated effect on the hard axis minor loop characteristic as 
a function of the width of a GMR structure with constant length. The strength 
of the magnetic anisotropy can be easily adapted to the sensor application: a 
strong anisotropy for applications where a wide linear range is advantageous, 
e.g. for rotary speed sensing or a vanishing anisotropy for applications which 
need the saturation magnetization to be obtained for low H-fields as it is nec- 
essary for angle sensors. Therefore, the GMR technology offers the possibility 
to cover the whole automotive magnetic sensor portfolio with just one GMR 
stack and slight variations of the sensor layer and/or GMR resistance layout. 
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magnetic field {mT) 

Fig. 9. Simulated hard axis minor loop as a function of GMR stripe width; 
stripe length of all structures is 250pm. 



4. Technology 

4.1. Monolithic Integration 

In contrast to AMR sensors where only one soft magnetic layer (e.g. permal- 
loy) of several 10 nm thickness is processed, the deposition of a GMR layer sys- 
tem with single layers in the sub nanometer range has to be well controlled in 
terms of absolute film thickness and homogeneity over the wafer. In the begin- 
ning, GMR stacks were deposited by MBE (molecular beam epitaxy) which is 
highly sophisticated and time consuming. In recent years, sputter tools with a 
highly accurate and reproducible thin film deposition as well as a high wafer 
throughput were made available on the commercial market. Therefore, GMR 
sensor processing became suitable for mass production with benefits for price 
and quality of the sensor products. 

Independently of the GMR stack composition, most of the processing of GMR 
sensors can be done using standard semiconductor equipment. The GMR mul- 
tilayer stack is deposited in a low temperature sputter process on top of a fully 
processed IC seperated by an insulating layer. 

All GMR stacks start with a specific seed layer defining the growth mode of 
the subsequent films and end up with a capping layer which prevents the 
underlying material from being oxidized. In order to avoid a degradation of the 
GMR sensor performance, the temperature budget of the following process 
steps have to be kept below a certain degradation limit which is in the range 
between 250° - 350° C for minutes - hours depending on the stack type. The 
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structuring of the GMR resistor can be done either by a dry etch or lift-off 
process. In the latter case, before the GMR sputter process a photo resist mask 
has to be generated on the substrate. 

Afterwards, a passivation with a GMR compatible temperature budget, e.g. by 
a sputter or specially developed low temperature CVD process, is deposited. 
After opening of the contact areas, a direct contact between GMR and the 
wiring metal is processed. Finally, depending on the sensor application, either 
a state-of-the-art magnetization process for the speed sensor or a special 
process for the local magnetization needed for angle sensing is applied. 

So far GMR sensors are available as discrete devices, i.e. without any signal 
processing on the same chip. In order to gain an utilizable sensor signal the 
connected signal processing chip may have following tasks: 

► Signal level amplification 

► Offset compensation 

► Gain compensation 

► Trigger threshold generation for speed sensors 

► A/D conversion for angular sensors 

► Angular calculation 

The signal processing chip has to be attached to the sensor chip e.g. by wire 
bonding or by a package-by-package solution. The results are higher induc- 
tive/capacitive parasitics, higher disturb signal sensitivity as well as increased 
leakage currents, thereby reducing the sensor performance. Furthermore, it 
could also be a major challenge to obtain a symmetric routing which is also 
important for excellent sensor properties. In addition, an input protection for 
the sensor chip is necessary. 

These drawbacks can be avoided by an integration of the GMR sensor on the 
IC chip. Results of the Infineon MR AM development reveal a high compatibil- 
ity of the integration of GMR materials in CMOS technology with respect to 
stack deposition, structuring, planarization processes and passivation. 

Furthermore, CMOS technology offers a variety of ways to provide conditioned 
surfaces which are very important to adjust the GMR growth condition and 
therefore, magnetic properties. In principle, there are two basic concepts for 
GMR integration into CMOS: 

► horizontal integration where the IC is beside the GMR sensor with no 
IC below and 

► vertical integration, putting the GMR device on places with IC beneath 
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The second concept gives the possibility of maximum integration density 
since no dedicated areas have to be provided. We have developed an integra- 
tion concept for GMR elements which is fully compatible to the CMOS pro- 
cessing with only minor changes of the IC base process. This is very advanta- 
geous because almost any CMOS and bipolar technology can be chosen for the 
IC. As a consequence, a transfer into different technologies like bipolar 
processes for higher voltage and current applications is easily possible. This 
covers a plurality of applications like power applications or advanced digital 
logic for signal processing. 



4.2 GMR Spin Valve thermal long Term Stability 

For automotive application the long term stability of the GMR stack is a key 
characteristic. As a monitor for changes in the GMR stack and magnetic prop- 
erties we will show examplarily the influence of thermal stress on the signal 
height of a GMR angle sensor. Fig. 9 demonstrates the relative GMR signal nor- 
malized to the initial value after high temperature storage between 175°C and 
275°C up to 2000h. 




storage time (h) 

Fig. 9. GMR signal as a function of storage temperature and time 

At the beginning, of the storage an increase of the GMR signal by up to 10% 
is observed as can be seen for 7"=225°C. For longer storage times the signal 
drops down again which is strongest pronounced at 7"=275°C. Origin is a 
strong decrease in the exchange bias strength between the natural anti-ferro- 
magnet and the pinned layer. As a result, the reference system is turned sig- 
nificantly by the rotating external field, thereby preventing a full parallel and 
anti-parallel alignment of the free and reference layer. The signal increase and 
decrease processes seem to be thermally activated. Assuming an Arrhenius 
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behaviour, an activation energy for the first signal increase of ^ 1.5 eV can be 
estimated which corresponds to a quite highly activated process. 

As a consequence, under typical automotive temperature condition of maxi- 
mum 150°C a signal change of ™1.5% after lOOOh is expected. Furthermore, 
a drop in the GMR signal is estimated to begin not before 240.000h at 150°C 
which is far beyond the required lifetime of automotive products. 

These results reveal the compatibility of the GMR technology to automotive 
requirements also regarding long term thermal stability. 



5 Incremental Position Sensors 

5.1 General Speed Sensor Requirements 

Usually toothed metal tone wheels are used as a target for speed sensors in 
powertrain applications. The general shape of the target wheels is not very 
predictable as it strongly depends on the application. Therefore very different 
magnetic signals may occur at a speed sensor. The wheel speed application 
uses widely pole wheels. The typical ambient operating temperature range for 
speed sensors is -40°C - + 150°C. Usually an air gap of up to 4.5 mm can be 
covered by Hall based speed sensors. GMR sensors allow now an increased air 
gap or better phase accuracy or repeatability. Table below shows general sen- 
sor demands for the specific application: 



Application 


Air gap 
[mm] 


Frequency 

[kHz] 


Repeatability demands 
[%] 


Wheel speed (ABS) 


0... 3.5 


0... 4.5 


Low 


Crankshaft 


0... 3.5 


0... 10 


High 


Transmission 


0... 5.5 


0 ... 12 


Medium 



Tab. 1 . Summarized requirements for incremental speed sensors in wheel 
speed, crankshaft and transmission application 



5.2 Comparison of the Technologies 

Variable reluctance (VR) sensors have been widely used in the past in all speed 
applications. Most important weak points on VR are the size and weight, which 
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limits the downsizing of modules. The output voltage of VR depends as well on 
the air gap and on the wheel speed which may cause additional efforts in the 
speed detection algorithm. 

All these drawbacks disappear by using Hall or AMR principle. The digital out- 
put of the devices is very predictable and just need a cheap digital input and 
only low signal conditioning. Advanced sensors like TLE4941/42 for example 
use a current interface of 7/14 mA which allows wire break detection and 
saves wiring costs. Integrated digital algorithms and improved analogue cir- 
cuitry enable new features like direction detection, vibration detection, air gap 
information and self calibration. Hall sensors allow a reasonable air gap and 
phase accuracy for most of the applications and can be fully integrated into the 
semiconductor standard production. In comparison the AMR devices are still 
very often not integrated and use a 2 or even 3 chips setup, which causes high- 
er production cost. 



Condition 


VR 


Hall 


AMR 


I-GMR 


Output signal 


analogue 


digital 


digital 


digital 


ECU signal conditioning 


Schmitt- 
trigger/ ADC 


digital 

input 


digital 

input 


digital 

input 


Module Size 


- 


+ 


+ 


++ 


Weight 


- 


+ 


+ 


+ 


Sensitivity 


+ 


0 


++ 


++ 


Temperature stability 


+ 


+ 


+ 


++ 


Zero speed & true power on 
function 


No 


Yes 


Yes 


Yes 


Direction detection and 
vibration detection 


No 


Yes 


Yes 


Yes 


Voltage supply needed 


No 


Yes 


Yes 


Yes 


Cost: IC + module + ECU 
conditioning 


0/- 


+ 


0 


+ 



Tab. 2. Comparison of different Sensor-Types (++ very good; + good; 0 
average; - weak; — very weak; Yes/No means that this function or 
condition is/isn't possible for this sensor type) 



Although discrete GMR have a better phase accuracy and allow a bigger air 
gap they struggle with the cost position. This can be improved by developing 
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an integrated GMR solution. The integrated GMR principle allows better phase 
accuracy and improved sensitivity but should not increase the total module 
costs against Hall sensors. Therefore some percentage of the sensitivity 
improvement may be used to reduce the costs of the magnetic circuit. So the 
usage of cheaper magnet materials and reduced size of magnet is possible. 

Furthermore the improved signal to noise ratio (SNR) reduces the demands on 
the internal signal conditioning circuitry. Signal amplification is getting less 
critical as well as the conversion from analog to digital world. All these points 
will cause further costs savings which should overcome the higher efforts to 
produce the GMR structures. 

Tab. 2 below shows the performance of I-GMR in reference to the other possi- 
ble sensor principles. 




Magn 



Field 




Fig. 10. Block diagram of Incremental Speed-Sensor. 



5.3 Architecture of an Integrated Sensor 

The general system realization of an integrated GMR sensor will be quite sim- 
ilar to the circuitry of an advanced differential Hall IC (Fig. 10). It contains a 
pair of GMR half bridges which are connected in a way to get a differential out- 
put signal. Then a summation point to compensate internal and external off- 
sets will follow. Here the signal path can be split into an analogue path where 
a threshold comparator (main comp) will control the output stage. Depending 
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on the application, different threshold concepts already available on Hall prod- 
ucts can be used. The other path is used to track the signal so that the cali- 
bration algorithm and the offset DAC can ensure that all internal and external 
offsets are compensated. 

The output signal is current modulated. The current levels 7 and 14 mA cor- 
respond to logic 'low' and 'high' level. This allows a connection of the sensor 
with only 2 wires. 

Of course, further GMR half bridges can be added to allow the direction detec- 
tion and vibration suppression. Only the output stage has to be extended to 
support a PWM or digital coded protocols. In further development, a fully dig- 
ital processing concept similar to the concept shown in the angle sensor sec- 
tion may be introduced. 



6 Angular Sensors 

6.1 Automotive Applications 

Angular Sensors can be found in several automotive applications in both 
engine and passenger compartment. The required angular accuracy strongly 
depends on the application. The better the accuracy, the more applications can 
be served. A few examples for values of the angular measurement range are 
listed in the table below. 



Application 


Measurement Range 


throttle-valve angle 


110° 


steering-wheel angle 


±2 x 360° 


pedal position 


30° 



Tab. 3. Ranges for different angular sensors applications 

6.2 Measurement Principle 

The GMR sensor circuit, like described in the previous technology section, 
changes its resistance according to the magnetic field components with 
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respect to the pinned layer of each GMR resistance. The change of the GMR 
resistance can be converted into a change of single-ended or differential volt- 
ages. These voltage-swings can be used to determine the correct angle of the 
magnetic field applied to the GMR sensor. The simplest way to detect the 
angle of the magnetic field is to calculate the angle out of the sine and cosine 
component of the magnetic field. 

Y 

a = arctan(— ) (5) 

As only the relation of X and Y is crucial, all amplitude effects, which influence 
both measurement paths in the same way, are automatically compensated. 




Fig. 11. Principle for 360° Measurement Range 



6.3 Signal Processing Path 

The two values X and Y are measured as voltages of two GMR bridges with 
perpendicular alignment of the pinned layers. X represents the cosine compo- 
nent and Y the sine component of the circular angle signal. Fig. 12 shows the 
orientation of the bridges on top if the signal processing chip. 

A point to pay attention to is the inaccuracy of the converted angle compo- 
nents X and y The main errors amplitude and offset can be corrected by 
arithmetical operations. The offset can be subtracted from the X and Y com- 
ponents. Unsymmetrical amplitude variations of the two signal paths can be 
eliminated by a multiplication to achieve the desired gain for angle calcula- 
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tions. Today, these corrections can easily be done by digital logic or even a pro- 
grammable microprocessor. This controller can be integrated in the signal pro- 
cessing chip. In this way the raw elliptic shaped GMR-outputs are corrected to 
a circle shaped figure to calculate the right angle by eliminating offset and 
amplitude variations. If a temperature measurement cell is integrated, addi- 
tionally temperature dependent effects can be compensated. This increases 
the accuracy also in a wide temperature range. 




Fig. 12. Orientation of the bridges 

The basic dataflow can be divided into two main parts, the analog signal area 
and the digital area. The analog area consists of the analog signal generation 
(GMR Signal Voltages), preparation for A/D-conversion and the A/D conversion 
itself. The digital part consists of component dependent digital signal process- 
ing, angle conversion and angle post-processing. 




Fig. 13. Basic GMR based Angle Measurement 

For the final 360° angle calculation 3 different methods can be used: 

► Fully programmed software algorithm (so-called 'Cordic') 

► Special hardware acceleration as additional logic module in a microcon- 
troller 
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► Look-up table with interpolation 

The analog parts include the blocks on the left side of the data path ending 
with the ADC, which converts the data to the digital domain. The angle cal- 
culation needs both components to calculate the angle, which is mapped after- 
wards to the right scale and angle offset. 

Basically it is useful to convert values to the digital domain as fast as possible, 
because temperature and aging effects of analog circuits cannot occur in digi- 
tal circuits. Only the calculation bit-width defines the calculation error in the 
digital domain. The signal speed is relatively low and the accuracy of each 
component is quite important. Therefore Sigma-Delta ADCs are most suitable 
in this application. With proper dimensioning of the ADCs, no preprocessing is 
required between the GMR sensor and the ADCs. This reduces the possible 
noise and error sources and the analog part becomes quite simple and short. 
Sigma-Delta ADCs generate a pulse-density 1 bit wide signal at a very high 
sampling frequency, which must be decimated and filtered to an appropriate 
useful signal-frequency. 



Benefit 


Linear Hall 


AMR 


GMR 


Sensor Size 


+ 


- 


+ 


Signal Level 


- 


0 


+ 


Sensitivity 


- 


+ 


+ 


Temperature Stability 


- 


0 


+ 


Weight 


- 


+ 


+ 


Module Size 


- 


+ 


+ 


Power consumption 


+ 


- 


+ 


System Costs 


+ 


- 


+ 


Angle Range 


120° 


180° 


360° 



Tab. 4. Comparison between Hall, AMR and GMR 



6.4 Comparison to other Technologies 

To compare different types (Tab. 4) of angular measurement, different charac- 
teristics of sensors must be kept in mind. For angular measurements, linear 
Hall uses a different effect to sense one sensor component. Depending on the 
angle, the flux density is modulated. To achieve this, difficult and expensive 
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magnetic circuits are required. From the application point of view the signal 
may be the same. Therefore we compare these magnetic sensor types with 
each other. For AMR and GMR applications, only a simple turning magnet is 
used, as only the field direction is the key parameter. A grading is done in the 
following way: + good, 0 average, - weak performance. For example in the 

row sensor size a + for the Hall type shows that a good size is possible as it 
can be built very small. 



7 Conclusion 

The GMR functional principle is based on scattering of conducting electrons of 
opposite spin at the interface of adjacent magnetized ferromagnetic layers 
with different direction. To exploit the potential of GMR sensors for a wide 
range of automotive applications, a cost effective, reproducible and reliable 
fabrication technology is required. Therefore, The integrated circuit (CMOS 
and Bipolar) processing has been taken as the starting point for the develop- 
ment. It offers the advantage that it is widely established and qualified for all 
kinds of automotive applications. It uses well-known processing techniques 
like sputtering and structuring of thin metal layers. Furthermore, monolithic 
integration of electronic circuitry for readout and conditioning of the sensor 
signal makes the system insensitive to parasitic effects of interconnects and 
electromagnetic coupling. 

In typical automotive applications the comparison with the established speed 
sensor types of variable reluctance. Hall and AMR showed significant advan- 
tages for the GMR technology. GMR sensors offer highest sensitivity and thus, 
allow performance gain by reduction of the jitter in the measured signals. On 
the other hand, an extension of the tolerable air gap between the sensor and 
the observed rotating targets is possible. The sensitivity gain may also be used 
to increase the allowable mechanical tolerances for the fixture of the sensor or 
to tolerate an operation using a less sophisticated back bias magnet. Both mod- 
ifications open a cost saving potential compared to competitive sensor tech- 
nologies. Furthermore, the GMR is the technology that exhibits the highest 
temperature stability. This allows increased performance by means of 
decreased tolerance over the automotive temperature range or reduced cost 
for time consuming temperature calibration measurements. 

Angle sensing is supported by spin valve GMR sensors which can be operated 
in a saturated mode. In this operational mode, the sensitivity for changes of 
the field strength disappears. The sensor is only sensitive for changes of the 
field direction with respect to the pinned layer magnetization direction. Using 
two GMR bridges with orthogonally directed pinned layers a sine and a cosine 
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signal can be measured and used for direct 360° angle calculation. The range 
of possible applications is extremely wide and will benefit from temperature 
stability of the GMR technology and from the significantly reduced require- 
ments for the magnetic circuit. 

Summarizing all technology comparisons and the analysis of various automo- 
tive applications, the GMR technology has been identified to be the most 
promising magnetic field sensing technology for the next decade. 
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Abstract 

The opportunities to integrate more and more functionality into a 
(micro)system-in-a-package (SiP) heavily call for advanced method- 
ologies in modeling and simulation for complete systems. For a 
microelectronics company like Infineon Technologies, the top-level 
simulation of our products, i.e. the above microsystems, is not a nice- 
to-have feature - it is a must! Iterating on the fabrication runs for 
design debugging simply is not feasible. Extending these simula- 
tions to application level can be accomplished and this paper shows 
how. An automotive drive application will serve as a demonstrator 
for that. 



1 Introduction 

Let's first take a look at the microsystem: it may comprise microcontrollers and 
power electronics. Behavioral modeling provides for sufficient simulation 
speed, while not sacrificing precision. For all analog and mixed-signal parts of 
the system, it is carried out in VHDL-AMS. This holds for electronics as well as 
for thermal processes, which always have to be taken into account when 
power electronics comes into play. The microcontroller is modeled in SystemC, 
which provides for an extremely fast, but still cycle-correct treatment. It takes 
real-life programs on assembly level and executes them as it is simulated. 

The application side is the electric motor with gear, load and a hall sensor for 
position feedback. Again, VHDL-AMS is adopted for behavioral modeling. This 
allows for the simulation of the complete application, comprising the microsys- 
tem as well as the surrounding electric motor and position using a commer- 
cially available simulator. This complete system typically simulates at a per- 
formance of 20 CPU-minutes per second real-time on a recent Linux worksta- 
tion, which allows for thorough system verification, software development and 
even concept assessment (in case the abstract behavioral models can be pro- 
vided in time, which should be no problem for a design with substantial lega- 
cy content). 
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2 System Modeling 

2.1 Introduction 

When talking about system modeling and simulation, often block-oriented sim- 
ulators are used like MATLAB/SIMULINK and similar tools. As the focus of a 
microelectronics company like Infineon is on electronic circuits, this kind of 
abstraction is too high, as even simple circuitry cannot reasonable be 
described in this way Nonetheless, MATLAB/SIMULINK component models 
may be linked into the kind of simulations, which are proposed in this paper. 




Fig. 1. Example drive system. 



In general, this leaves us with two approaches for the modeling of systems 
spreading over more than one physical domain: simulator coupling and model 
transformation: 

► Simulator coupling 

In our case, the simulator coupling approach would have to bind togeth- 
er simulators for analog electronics, digital electronics, software, 
mechanics, thermal issues and magnetics. Even if it was possible to 
solve all the interface issues, we would end up with a big hunk of soft- 
ware, which would be extremely difficult to manage and configure. 
Moreover, the combination of several analog solvers would easily lead 
to very nasty convergence problems. 

► Model transformation 

The idea behind model transformation is to provide all models in one 
common form, so that they are simulatable by one solver. 

We chose the second approach in which all domains are covered by one solver. 
Actually, we have got two solvers - one for analog and one for digital - which 
is due to the nature of the underlying descriptions. This co-simulation of the 
analog and digital domain has been a topic in microelectronics for twenty 
years now. The tools on the market are mature and reliable. The language to 
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formulate the analog models is VHDL-AMS. For the digital part we chose 
SystemC. 



2.2 Power Electronics 

The typical components of the power electronics are the following: 

► Power switches, highside and/or lowside 

► Motor bridges and bridge drivers 

► Voltage and current regulators 

► Transceivers 

The basic behavior of these kind of components is determined by the behavior 
of the respective power MOSFETs and their gate drivers, several protection 
features (e.g. over- temperature, over-current, etc.), several diagnosis features 
and the environment, comprising of the loads (e.g. electric motors, bus lines 
etc.) and the attached microcontroller. With the reduction of the number of 
chips in an electronic control unit (ECU) and the resulting need for higher inte- 
gration, many of the above features are integrated into one chip or at least into 
one package. 

The modeling of power electronics - as any other kind of modeling - always 
requires a reasonable balance between precision and model complexity. 
Providing for more precision, i.e. adding more related physical effects into for 
instance the power MOSFET, directly leads to a longer simulation time. The art 
in it is to stay as precise as necessary and as abstract as possible. In our case, 
this means that the power MOSFET is to be modeled in higher precision and 
on an abstraction level, which is comparable with the device models for the 
circuit simulation. On the other hand, the protection features or the diagnosis 
features and even the gate drivers are modeled in a relative abstract fashion, 
as their impact on the outside world is limited or their contribution can be 
described in an event-oriented or even digital manner. 



Drain 




Fig. 2. Electro-thermal DMOS model. 
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When talking about power electronics, we also have to talk about self-heating 
and in general thermal issues. Even more, the circular dependency between 
the electrical and the thermal behavior of a power transistor has to be taken 
into account. This is due to the drain current which causes the self-heating and 
the resulting junction temperature which influences the drain current. Fig. 2 
shows a DMOS model with attached network of thermal capacitances and 
thermal resistances. 



2.3 Embedded Microcontroller and Software 

The microcontroller's task is to run the embedded software and to provide for 
the interfacing to the outside world through its peripherals. Again, the issue of 
precision and abstraction comes up. The first question to answer for a micro- 
controller model is that of the underlying time base. In all practical cases up to 
now, it turned out, that microcontroller's clock cycle is precise enough for a 
reasonable system simulations. On the other hand, it allows for a simulation 
speed that its one or more orders of magnitude above the gate level simula- 
tions, which would form the next level of precision. 

For the microcontroller in our case - a state-of-the-art 8-Bit controller - we pro- 
vided for a SystemC model, which was attached to the VHDL-AMS models for 
the rest of the system. This was accomplished by using the standard SystemC 
interface as provided by the vendor of the commercial VHDL-AMS simulator. 
The modeling strategy again follows the rationale of as precise as necessary 
and as abstract as possible. As the underlying idea is not to enhance the 
respective microcontroller, it is sufficient to provide for a model that is only 
valid at its external interfaces. Basically this model is a command interpreter 
which reads the current assembler statement and runs the respective actions 
on its memory map. The models for the peripherals are attached to this com- 
mand interpreter and bind it to the outer world. The embedded software itself 
is not modeled. At initialization time, it is loaded into the respective memory 
model. Then it is executed as the microcontroller model is simulated. 

As the SystemC model just provides for digital behavior, an analog shell was 
attached to it, to accomplish for correct electrical behavior. This turned out to 
be extremely helpful, as mismatches can be detected easily in this manner. 
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2.4 Mechanics 
2.4.1 Introduction 

The demonstrator, which is driven by the microsystem, is an electric window 
winder like it is widely used in car doors, see Fig. 3. Such a macro-system con- 
sists of different components which comprise different domains, like electrical 
(DC-motor with Hall-sensor), mechanical (gear, cable, window, guide rails), 
electromechanical (DC-motor), electromagnetic (DC-motor, Hall sensor). As an 
example for the basic effect approach we have a closer look at the mechanical 
effects, which are: 

► Inertia: gear wheels (rotational), window (translational) 

► Static and dynamic Friction: gear, guide rails 

► Transmission (rotation-rotation, rotation-translation) and clearance in a 
gear Note: the cable is implemented as gear with rotational input and 
translational output. This approach serves to convert the rotation into a 
translational movement. 

► Mechanical stop: upper and lower limit, "children hand" 




Fig. 3, 



Automotive window winder system 
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2.4.2 Modeling and Implementation of the Mechanical Parts 

The electro-mechanical and electro-magnetic parts of the envisaged demon- 
strator is based on the modeling of basic physical effects which occur in or 
between single components of the mechatronic sub-system. The advantage of 
this kind of modeling is the high reusability of already modeled and validated 
effects. This approach benefits from the option that the designer is able to 
decide whether an effect will be considered or not and so influences the sim- 
ulation precision and speed. Another advantage is that no knowledge of the 
model equation of the complete system is necessary - it is automatically build 
inside the simulator by combining the needed effect models. 

In VHDL-AMS, we are not bound to given variables. This is an advantage, since 
we can implement variable types of non electric domains, which we want to 
simulate. VHDL-AMS offers outstanding possibilities for the definition of user- 
defined variable types. Tab. 1 shows the definitions of the flow and potential 
variables like they are recommended by the IEEE and for this system are 
excellent suitable. 



Definition of flow and potential variables 


Variable type 


Electrical 

network 


Rotation 


Translation 


Through 

flow-variable 


Current I 


Torque T 


Force F 


Across 

potential-variable 


Voltage V 


Angle qp 


Displacement x 



Tab. 1 . IEEE recommended definition of flow and potential variables 



The restrictive port definition guarantees both the compatibility of the effect 
models among themselves, if one wants to interconnect them, and a simple 
expandability of the model library. 

A simple example of physical effect implementation (the model of the gear 
transmission rotation/rotation without static friction or clearance) in VHDL- 
AMS is shown in Listing 1. The model contains two formulas: 

► relation between the output torque and the input torque: 

^out “ ”^"in 9r V 

► relation between the angle of the input gear wheel and the output gear 

wheel: 6 0ut = -0 ia / g r 
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With T the torque, g the gear ratio, 77 the efficiency factor and 6 the rotation 
angle. 



01 . 


LIBRARY IEEE; 












02. 


USE IEEE. MATH REAL. ALL; 












03. 


USE IEEE. MECHANICAL_SYSTEMS. ALL; 










04 . 


ENTITY gear_rot_rot IS 












05. 


GENERIC ( g_r : REAL := 1. 


0; -- gear ratio 










06. 


eta : REAL := 1. 


0); -- degree of effectiveness 


(0 


<= eta <= 1) 


07. 














08. 


PORT (TERMINAL rot_p_in. 


rot_n_in. 










09. 


rot_p_out , 


rot_n_out : ROTATIONAL ) ; 










10. 


END ENTITY gear_rot_rot ; 












11. 














12 . 


ARCHITECTURE simple OF gear_rot_rot IS 










13. 


QUANTITY theta_in ACROSS 


T_in THROUGH rot_p_in 


TO 


rot 


n 


in; 


14 . 


QUANTITY theta_OUt ACROSS 


T out THROUGH rot_p_out 


TO 


rot 


n 


out; 


15. 














16. 


BEGIN 












17. 


T_out == - T_in * g_r 


* eta; 










18. 


theta out == - theta in / 


g_ r ; 










19. 


END ARCHITECTURE simple; 













List. 1. VHDL-AMS implementation of the rotational/rotational gear trans- 
mission 



2.4.3 Modeling and Simulation of the Mechanical Sub-system 

For modeling the mechanical sub-system, the available models have to be 
interconnected. Due to the defined flow and potential variables the related 
models are port compatible. A symbol was created to each VHDL-AMS model, 
which can be interconnected then with another symbol in the schematic edi- 
tor. The creation of the netlist of the system can be done automatically with 
the net list generator from the schematic editor. For exemplification the 
schematic of the sub-system is shown in Fig. 4. 




Fig. 4. Schematic of the automotive window winder sub-system with test- 
bench 
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After the fundamental physical effects were identified, implemented in VHDL- 
AMS and combined to the electro-magneto-mechanical macro system, the 
model of the mechatronic sub-system is ready to simulate. 



3 Simulation Results and Conclusion 

The model setup as described above has been simulated using 12 tests and 
related 12 embedded test programs. These were developed to verify the 
design of the (micro)system-in-a-package as described above. Virtually the 
complete functionality of the product can be assessed in its environment. 
These simulations turned out to be a cornerstone in the SiP-verification, as 
they helped to avoid a redesign. The simulation speed is in the range of 20 
CPU-minutes per second real-time, which allows to accomplish extensive veri- 
fication runs. 

The limitation of this kind of simulations lies in the fact, that only those effects 
are taken into account, which have been introduced into the models. So if we 
know, that self-heating is an issue, we can take care of that. But if we do not 
take into account some certain effect, the system may later fail because of 
that, though the simulations were successful. In other words, this means that 
we get some information on whether the system can work under certain con- 
ditions rather than whether it will work under all conditions. 

The shown drive system served as a demonstrator to assess the opportunities 
of complete system simulations. With that, the neuralgic scenarios can be 
assessed before tapeout of the chips. Even more, the complete controllability 
and observability of a simulation helps a lot in quickly tracking down the prob- 
lems. All in all, the proposed approach will help to jointly get a much better 
understanding on the behavior of an even complex SiP in its environment. 
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Abstract 

Innovative packaging technologies and power aware design enable 
system shrinking considering automotive requirements. During the 
design process the coupling of very narrow positioned components 
and the interdependence of hardware, software, and packaging have 
to be considered. The relevant miniaturisation aspects are discussed 
on the exemplary application of a tire pressure monitoring system. 



1 Introduction 

Miniaturized wireless sensors become more and more important. They are 
increasingly applied to everyday objects, e.g. for tire pressure monitoring to 
enhance transport safety. Over 440 thousands people were injured or killed by 
traffic accidents on Germany's roads in 2004 [1]. Thereby maladjusted or 
improper filled tire equipments cause 30 percent of technical problems result- 
ing in traffic accidents with personal injury [2]. Next to safety awareness sat- 
isfaction, correct pressure improves fuel economy and pollution control. 
Random inspection by NHTSA mentioned that pressure is erratically checked 
by drivers and over a quarter of tested vehicles have at least one improper 
filled tire [3]. As a result of this study the US American legislator passed a law 
[4], which claims to equip all new vehicles with tire pressure monitoring sys- 
tems after September 1 st , 2007. This measure affects about 8 million passen- 
ger cars per year [6] and implies a huge market for self-sufficient micro sys- 
tems, while Germany exports about a half million cars per year into the United 
States [7]. 

Today's rapid development in packaging technologies enables direct tire pres- 
sure monitoring systems (TPMS) by additional downscaling of self-sufficient 
wireless sensors from some cubic centimetre to less than 1 cm. They allow effi- 
cient solutions for TPMS, which requires low weighted self-sufficient sensors 
to reduce out-of-balance forces. Furthermore the lowering of substrate area 
reduces mechanical stress applied to the electronic device. But not only tire 
pressure monitoring systems can benefit from miniaturized wireless sensors. 
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Tiny wireless ultrasonic sensors and cameras mounted on wing mirrors are 
imaginable to monitor blind angles and to be integrated into parking distance 
control systems. Ongoing system miniaturization at the Fraunhofer Institute 
for Reliability and Microintegration (IZM) targets self-sufficient wireless sen- 
sors with an edge length of 6 mm. 

During the design process, the coupling of very narrow positioned components 
have to be considered in ultra dense 3D packaging more then in conventional 
2D printed circuit board design. And also the low-power design benefits the 
overall system volume and dimensions due to usage of smaller batteries. So the 
development of tiny low weighted and mechanical robust wireless sensor sys- 
tems needs the reflection of multiple disciplines: hardware, software and pack- 
aging. Thereby exploitation of battery effects allows a reduction of battery vol- 
ume. The relevant miniaturisation aspects are discussed on the exemplary 
application of a wireless tire pressure sensor unit. 

In the next three sections, we will discuss the degrees of freedom in selecting 
components and packaging of miniaturized self-sufficient sensors like wireless 
tire pressure sensors. Beginning with the exemplary requirements on TPMS, 
we will introduce selection of components and interconnection technology as 
part of design flow to achieve miniaturized wireless sensors. Different design 
options are traded off to meet demands. Finally we present prototypes of net- 
work capable wireless sensors as well as sensor modules for a direct TPMS 
developed for Global Dynamix AG. 



2 Requirements 

Requirements on wireless sensors are defined by its application and lead the 
designer through the design space. Because we use it as an example, we want 
to introduce general requirements on tire pressure monitoring system: 

Reliability: If a tire pressure system or another assistant system is at driver's 
disposal, it is trusted. Hence, reliability is intended by packaging. Wireless sen- 
sors mounted in rotating wheels are caught in crossfire of harsh environmen- 
tal influences. Vibrations and acceleration forces necessitate mechanical 
robustness. A wide temperature range and thermal shock resistance are 
required by meteorological condition as well as heating by full breaking. 
Sensor units must be shielded against electro magnetic irradiation and have to 
be resistant against chemical substances, i.e. salt-water, lubricating oil, sul- 
phurous lotions, or cleaning supplies. In all cases, the integrity of sampled 
pressure values is a necessity and the driver has to be informed when a mal- 



Miniaturized Wireless Sensors for Automotive Applications I 355 



function is detected. Reliability also addresses fault-tolerant system behaviour 
(implemented in hard- or software), which is not considered in this paper due 
to limited space. 

Low unit cost: Designed for mass marked, unit price has to be low. Reducing 
the number of electronic devices decreases component costs, and decreases 
the overall system weight and volume. Integrating analogue acquisition com- 
ponents, digital circuitry for computation and RF front-end for wireless com- 
munication into a single package allows a very compact single chip solution. 
Yield is primarily a function of die size. Therefore multi-chip solutions are 
sometimes preferred because of cost-effectiveness. However, the system 
requires the inexpensive integration of heterogeneous components, i.e. ICs, 
sensors, crystals, antenna, and passives. 




unit. 



Low installation and maintenance effort: Installation of tire pressure moni- 
toring system should not influence established factoring processes. Amount of 
cabling should be as low as possible. Therefore systems with a single receiver 
are favoured over antennas in each wheel house. Reutilisation of keyless-entry 
board unit for tire pressure monitoring reduces installation effort and system 
costs. Exchanging batteries of tire pressure sensor units mounted in air cham- 
ber is time consuming and expensive. Therefore long operation time must be 
granted and a tire pressure sensor should work over a tire's life time by utiliz- 
ing an ultra low power design. 

In this paper we concentrate on techniques to achieve low system dimension 
and weight. 
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3 Selecting Components 

Fig. 1 shows a block diagram of a wireless tire pressure sensor. It can be easi- 
ly adapted to other wireless sensors applications: Temperature and pressure 
are acquired by sensors. After conditioning digitized signals are preprocessed 
and assembled into information packets, which are transmitted to the board 
unit using a radio link. The radio link can be used in opposite direction to 
receive calibration and configuration data. All components are powered by a 
power supply unit. 

In the field of system miniaturization, the main question is which components 
mainly determine the final dimensions of the whole system. The optimization 
focus during the hardware development is to be put accordingly. Components 
like batteries, capacitances, high quality filters, crystals, and antennas nor- 
mally dominate the volume of a wireless sensor node below a cubic centime- 
tre. Several trade-offs often allow to adjust the requirements of a single com- 
ponent and to change volume share for a minimal size of the whole sensor 
node. For instance, a larger battery allows a smaller antenna with a reduced 
efficiency and vice versa. Our first design stage is to consider the manner, in 
which functionality will be realized. The subsections below discuss design 
options according to the requirements mentioned above, and keeping power 
consumption as well as system size in mind to achieve a long operation time 
and high mechanical robustness. 



3.1 Sensors 

Temperature, acceleration forces and supply voltage are reasonable measure- 
ments for TPMS next to obvious pressure. Temperature monitoring is neces- 
sary for temperature compensation and allows warning on overheated tire. 
Vibrations are indicators for a moving vehicle and can be used to wake up the 
TPMS from very deep sleep mode, while detection of low supply voltage is 
necessary to warn the driver about imminent sensor malfunction. The size and 
power consumption differs very much according to the type of sensor tech- 
nology. With declining smaller sensor dimensions, the physical connection to 
the measuring medium also declines, so that miniaturization can considerably 
influence the precision. In this section we focus upon pressure and tempera- 
ture measurement to avoid going beyond the scope of this paper. The selection 
of any transduction techniques affects the complexity and size of read-out cir- 
cuitry. 
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Nominal pressure [kPa] 
Temperature range [°C] 

Signal proportionality 

Electrical impedance 

Signal drift 

Signal noise 

Offset signal 

Relative signal change 

Power consumption 

Influence of parasitics 

Read-out circuitry 

Overload-ability [nominal pressure] 

Temperature sensitivity [%/K] 



Piezo-resistive 


Capacitive 


10-1000 


10-1000 


-40- +125 


-40 - +125 


displacement 


1 / displacement 


low 


high 


high 


low 


low 


moderate 


moderate 


small 


small (~2%) 


large (-50%) 


high 


low 


low 


high 


simple 


complicated 


5x 


lOOx 


-0,15 


-0,01 



Tab. 1. Comparison between piezo-resistive and capacitive pressure trans- 
ducers [5] 



Pressure: Miscellaneous pressure transducers are available on market: resis- 
tive wire strains, piezo-resistive and capacitive pressure transducers. 
Transducers used for TPMS operate in range between 100 and 500 kPa and 
must be insensible to acceleration forces. The first one is unsuitable, because 
resistive wire strains are less appropriate to pressures below 500 kPa appear- 
ing in tires and their miniaturization potential is limited [5]. Piezo-resistive 
pressure transducers change their resistance when deformed by applied pres- 
sure, while capacitive pressure transducers alter their capacity with distance 
of electrodes. Tab. 1 compares both pressure transducer types. Capacitive 
transducers are less sensitive to temperature but have high requirements on 
read-out circuitry. Influence of parasitic capacities is high due to measuring 
capacities in the range of 10 pF. Their low power consumption is less weight- 
ed, because sensors are low frequently used and activated for a very short 
duration. Specific package have to be attended to reduce disturbances by ther- 
mal mismatch between chip and package, whereas mechanical restraint due 
to fixed assembly of chip and package can be seen as reason. 

Temperature: A sensor module for TPMS must operate in temperatures 
between -40 °C and 125°C. Suitable transduction techniques are realizable in 
compact form. Metal resistors, negative and positive temperature coefficient 
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thermistors, silicon-spreading-resistance, and integrated sensors allow simple 
read-out circuits, while crystal sensors offer higher accuracy [5]. Tire pressure 
monitoring systems do not require high temperature accuracy 



3.2 Data Processing and Device Control 

Signal conditioning and communication management is done by digital back- 
end. Pre-processing allows the reduction of transmitted data in order to save 
energy The corresponding algorithms can partly be implemented energy effi- 
ciently as hardware or more flexible as software (Fig. 4). Hybrid approaches 
will often represent the most suitable solutions. Low level functions like the 
signal acquisition and base band processing are often realized via application 
specific, integrated circuits. This represents a benefit for computation-devices 
regarding computation speed and size. Depending on production volume and 
required flexibility, high level functions are implemented either in hardware 
or in software. In the case of our TPMS standard, microcontrollers with hard- 
ware support for data acquisition are used to achieve a short time to market. 
However, power consumption and battery size benefiting application specific 
integrated circuits are predicted due to high TPMS quantities. 

Next to signal conditioning and communication management the digital back- 
end implements the power saving policy. An efficient power adaptation for sin- 
gle components allows a considerable reduction of size determined by energy 
supply. The standard measure is to switch off all components which are not in 
operation. In case of TPMS the impact of power down current increases due to 
very low duty cycles (below 0.1%). The microcontroller requires an operation 
mode with a very low leakage current and a possibility for periodic wake-up. 
U.S. Federal Motor Vehicle Safety Standards No. 138 [4] allows relaxed duty 
cycles for TPMS by an acceptance of reaction time up to 20 minutes. 



3.3 Radio Interface 

The design of radio interface and communication protocol is driven by follow- 
ing intentions: a) reliability of communication channel, b) size and weight of 
required components, c) power consumption. The selection of carrier frequen- 
cy plays a key role. Higher carrier frequencies allow smaller antennas and 
higher data rates, whereas longer ranges and less reflection on surfaces are 
possible with lower transmission frequencies. Frequencies selection is limited 
by national frequency assignment plans. We choose 868/915 MHz for TPMS 
due to antenna size and transmit power consumption balance. 
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Antenna design is not only influenced by the limited space but also by shield- 
ing effects of the electrical circuitry and the power supply. Apart from the 
development of an efficient antenna, the optimum for the whole system can 
be a smaller antenna of reduced efficiency where the saved volume allows a 
higher energy buffer capacity. In the case of the proposed TPMS lower battery 
weight has a higher priority then antenna size. The system has to work with 
a single antenna attached to the board unit. It is not satisfying to mount 
antenna in each wheel house due to wiring effort, which provides optimization 
criterion for antenna design in case of TPMS. 

A further task of the radio interface is the reception of configuration data. 
Reconfigure ability of alert thresholds, duty-cycles and carrier frequency 
allows a very flexible system which can be used in different vehicles and coun- 
tries. This makes integration of a receiver necessary, which also enables 
whole-system calibration after module assembly. The communication protocol 
has to reduce idle-listening, i.e. power-waste of an active receiver without 
incoming messages. 



3.4 Clock Generating Components 

Because clock generating components can take a high volume share, they are 
discussed separately. For the clock generating components a decision had to be 
made which frequency tolerances could be admitted. On-chip RC oscillators of 
a very low volume show a drift of 3 - 25% due to aging and temperature 
dependency. Frequency of larger crystals vary below of 0.1%. The clock source 
for radio frequency synthesis is most critical to keep radio frequency drift low. 
This clock source must be quite temperature and long term stable, because the 
tire pressure sensor is exposed to wide temperature range over several years. 
It is only possible to apply a frequency trimming factor in dependence of tem- 
perature if aging effects are negligible. In that case the avoidance of expensive 
temperature stable crystals with relative high volume share is possible. 



3.5 Power Supply 

An essential component of the self-sufficient wireless sensors is their own 
energy supply, which strongly determines size. The energy supply of tire pres- 
sure sensors is often based on electrochemical batteries due their relative high 
energy density and relative low costs. The selection of appropriate micro bat- 
teries can raise issues. Not only high energy densities for small volumes, but 
also the nominal voltage, the self discharge current, and the maximal load 
have to be considered for the choice of material. For instance, zinc air cells 



360 I Components and Generic Sensor Technologies 

with a very high energy density (Fig. 2) can be used only for a few weeks due 
to their high self-discharge current. On the other hand, lithium button cells 
can only drive relatively low currents, which can cause non-tolerable voltage 
drops in transmission phases. Here, electrode area is a major factor. 

Furthermore, the estimation of the necessary battery volume for a given oper- 
ation time is quite complex. The capacity of a small battery decreases rapidly 
at higher loads, whereas the capacity increases during intervals of reduced 
currents. This is known as the rated-capacity effect and the recovery effect. 
Additionally temperature dependency of chemical reactions, aging of battery 
caused by side reactions have to be considered - especially in case of long 
operating TPMS. Hence, determination of required battery size depends mas- 
sively on the load profile and climatic environment. 



Fig. 2. Energy capacity of some batteries [9]. 

Indeed batteries have the lowest primer costs, but we have to think about the 
huge amount of batteries, which have to be disposed as hazardous waste after 
TPMS operation time ends. Some applications allow a replacement of large 
energy buffers by energy harvesting using movement, heat, radiation or bio- 
chemistry. The size of energy converters depends on energy demand and 
available intensity. Energy buffers will be necessary for most cases, because 
the harvested energy will rarely satisfy the total power consumption at any- 
time. Piezoelectric devices are imaginable in TPMS to benefit from vibration, 
but they require huge vibration pick-ups and oscillating weight, which increas- 
es wheel unbalance. An additional option is the usage of wireless charging 
techniques to enhance the system life time. Besides electrical charging via 
radio antenna or inductive coupling, the same principles of energy harvesting 
can be taken into account depending on coupling factor, shielding, and direc- 
tivity. Charging by inductive coupling adds unwanted wiring effort in case of 
TPMS, because coils have to be mounted in each wheel house. 




Volume [mm 1 ] 
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4 Integration Technology Selection 

As have been shown above, a system like TPMS consist of several heteroge- 
neous components. Design methodologies and reliable packaging technologies 
of heterogeneous components are research objects of Fraunhofer Institute 
Reliability and Microintegration and Technical University of Berlin. 




2004 2005 2008 2007 2008 2009 



Fig. 3. SiP market projection, millions of units [10] 

The common planar On-Board integration techniques like SMT (Surface 
Mounted Technology) allow low cost realization. But the increasing system 
complexity and high miniaturization demand for the automotive applications 
require another integration technologies, which have to be applied to achieve 
a size constrains. On-Chip integration of heterogeneous elements is challeng- 
ing from design as well as from technological point of view. A promising solu- 
tion for achieving highest integration density in smallest volume has been 
seen in vertical integration of heterogeneous components as a System-in- 
Package (SiP) (Fig. 5). This integration approach is well established in the area 
of wireless communication and it is also gaining for automotive market more 
and more in importance (Fig. 3, [10]). 

The selection of the optimal integration technology as well as geometrical 
arrangement of components and preparing of the manufacturing data are 
obvious parts of the physical system design. It begins after the previously 
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described functional units are defined und their electrical schematic is fin- 
ished. In the beginning of the physical design, the component set (ICs-bare 
dice, passives in their values etc.) for the realization of TPMS and their electri- 
cal interconnection (net list) are defined by the schematic. Within the 2.5D 
integration, where the functional layers are vertically integrated on each 
other, the physical design could be subdivided into two stages [8]: 

► Global Layout, which can be abstracted away 
from any certain technology, 

► Detailed Layout/implementation, which is obviously 
based on definite technology. 




beginning, b) after partitioning, c) after placement and routing 



In analogy to PCB/MCM- and IC-Design, both stages consist of three main 
steps: Partitioning, Placement and Routing. The aim of global partitioning is to 
group the components and to allocate them on certain vertical layers. Global 
placement means the arrangement of layers in stack, and global routing is in 
this case the optimization of the electrical wiring/nets between the layers (Fig. 
4). All these steps can be formulated as a mathematical problem with a num- 
ber of boundary conditions and objectives, such as floor plan and volume mini- 
mization for partitioning, minimization of wire length in the placement, the re- 
duction of nodes quantity in the routing, testability of the layers and of the 
complete system as a general task etc. 

Whilst a designer of "traditional" planar applications such as PCB or MCM has 
several clear technological criteria to make a decision regarding substrate and 
interconnection technology (e.g., wiring density, via types, the number of lay- 
ers of laminated, thin film or ceramic substrates, interconnect pitch, cf. Tab. 2 
and 4), there is no clearly defined criteria for choosing optimal SiP technology 
for vertical integration. The system designer is faced with many different 
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technological solutions presented by several industrial and research institu- 
tions. The currently available and practicable technological 2,5D SiP solutions 
for realization of highly miniaturized wireless sensor systems such as TMPS 
can be exemplary subdivided in four main groups [8] (Fig. 5): 




Fig. 5. Four main integration schemata and application examples of 2,5D 
SiP 

1. Systems, where no signal redistribution in vertical wiring (In this case, 
it can be called vertical interconnect - VIC) and lower signal redistrib- 
ution level in lateral wiring possible, so the routing can be down only in 
lateral level (mostly represented by wire bonded die stakes, e.g. Sharp), 

2. Systems with no signal redistribution in VICs and high signal redistrib- 
ution / routing capability in lateral levels (represented by solutions with 
functional layer on stacked modules e.g. North), 

3. Systems, where signal redistribution level in vertical and lateral wiring 
is high and approximately equal (represented by solutions with compo- 
nents on the folded flexible substrate, e.g. Tessera), 

4. Systems with medium and approximately equal redistribution level in 
vertical und lateral wiring, represented by stacked and molded devices 
with connecting metallization on the surface/sidewalls (3D Plus, Irvine). 

For the compare and choice of optimal technology for the realization of TPMS 
at the beginning of the detailed layout/implementation stage, the designer 
needs objective criteria, as they are usual in the two dimensional design flow 
(Tab. 2 and 3). Tab. 4 demonstrates partly a parameter set for the four labeled 
technology groups. The parameter set is currently been explored at 
Fraunhofer IZM and practically verified by the prototypes, which are dis- 
cussed in the section 4. The parameter provide not only key data for a quali- 
fied comparison and selection of integration technologies, they can also be 
used for the constrained placement of the components in 2,5D-Room in global 
layout stage. The available CAD tools for PCB-, MCM- and Packaging-Design 
include highly automated autoplacer and autorouter for the two-dimensional 
applications. But for vertical SiP- In-teg-ration of heterogeneous systems like 
TPMS, the designer still places components manually over the number of lay- 
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ers. The Fraunhofer IZM is working currently in cooperation with Fraunhofer 
ITWM on the novel 2,5D auto placement procedure, which is based on the vol- 
ume optimization algorithms constrained by investigated technological param- 
eter, targeting higher design effectiveness for TPMS- and further automotive 
applications. Furthermore, the automotive applications require very smart and 
robust concepts for the system encapsulation. In the traditional packaging 
design, the system mol-ding issues are often been seen as an additional post 
design step. In the case of 2.5D integration for automotive area, the encapsu- 
lation concept influents the design flow in early stage for example during layer 
planning and com-po-nents placement. Fraunhofer IZM investigates also mold- 
ing technologies for the application environments with high temperature and 
mechanical stress, as they are usual in the automotive area. 





MCM-L 


MCM-C 


MCM-D 




HDI 


PCB 

Standard 


Ceramics 


Thin Film 


Line width [jum] 


50. ..75 


125 


75.. 100 


10 


Line space [pm] 


50. ..75 


125 


250 


10 


Via land diameter [pm] 


100. ..225 


650 


200 


30 


Number of layers 


8. ..10 


8. ..30 


15. ..30 


2. ..5 


Dielectric constant [mm 2 ] 


2, 3... 4.7 


4,7 


6...10 


2, 7. ..3, 5 


Material 


FR4 


FR4 


Alumina 


Si, 

Metal... 


Price appoxim. 


Medium 


Lowest 

(cents/cm 2 ) 


Medium 


High 

($/cm 2 ) 



Tab. 2. Design relevant parameters for laminated (MCM-L), ceramic 
(MCM-C) and thin film (MCM-D) substrate technologies [8] 





Wire Bond 


Flip Chip 


TAB 1 


Min. Pad Pitch [pm] Die 


50 


100 ... 120 


60 


Substrate 


120 


100 ... 120 


200 


Mounting 


serial 


parallel 


serial/par. 


Electrical Performance L [nH] 


1 ... 5 


0.06 ... 0.2 


1 ... 3 


C [pF] 


0.2 ... 0.6 


0.02 ... 0.03 


0.2 ... 0.6 


Mechanical Protection 


glob top 


underfill 


none 



Tab. 3. Design relevant parameters for different interconnect technologies 
[ 8 ] 
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Stacked 

Dice 


Stacked 

Modules 


Folded Flex 


Moulded 

Devices 


Redistribution 


Vertical 


- 


- 


high 


medium 


capability 


Lateral 


low 


high 


high 


medium 


Passive Integration 


Discrete 


low 


high 


high 


high 


Capability 


Embedded 


low 


high 


medium 


medium 


Wire Length 


WL 


high 


low 


high 


medium 


No. Functional Layer 


L 


up to 5 


7 


10 


32 


Layer Thickness 


Layer Gap G [pm] 


negligible 


100 ... 1200 


200 ... 1200 


50 ... 600 


L—G+C 


Carrier C [pm] 


200 ... 600 


50 ... 1200 


20 ... 100 


50 ... 200 


Vertical Interconnect 
Density 


[l/mm 2 mm] 


0.5; 

~f(G) 


0.5 ... 12; 
~f(G) 


5 ... 30 


10 ... 50 



Tab. 4. Parameter set for labelled SiP technology groups 



5 Prototypes 

For evaluation of high integration packaging techniques tiny wireless sensor 
are implemented on flexible substrates (Fig. 6a). These sensors measures 
brightness and temperature and interact in multi-hop sensor networks. All 
components are integrated in volume of 1 cm 3 . A self-sufficient tire pressure 
sensor unit developed for Global Dynamix AG is shown in figure 6b. They 
allow the reconfiguration of the tire pressure sensor module using radio com- 
munication like measure intervals, thresholds and radio frequency. The system 
uses 868/915 MHz-ISM-band and allows the usage of a single antenna inte- 
grated in the board unit. 




Fig. 6. a) Components of wireless sensor mounted on folded flex 

b) Electronic of a tire pressure monitoring sensor developed for 
Global Dynamix AG 
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6 Conclusion and Future Work 

In this paper we discussed the degrees of freedom in designing miniaturized 
self-sufficient sensors considering a TPMS as an example. Especially tire pres- 
sure sensors electronics can be extended to equip a tire with intelligence and 
to allow vehicle adaptation to weather and road condition. The tire ID in future 
will prevent the usage of ineligible tires in dependence on vehicle type, road 
and weather conditions. This addresses a further main source of accidents 
with damage to persons. Thereby miniaturization of application specific autar- 
kic sensors requires system knowledge and multidisciplinary overview due to 
interdependences of different domains. Today's advances in packaging and 
manufacturing technology allow the enhancement of the miniaturization level, 
which enables the integration of reliable self-sufficient sensors into objects like 
valves. Further power efficiency is needed to achieve long operating light- 
weight systems. 
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Abstract 

The MID (molded interconnect devices) technology provides selec- 
tive metallization in three dimensions on injection molded thermo- 
plastic substrates and housings. Different technologies can be used 
to metallize plastic parts. Automotive applications for MID technolo- 
gy are pressure and flow sensors for engine management, air condi- 
tioning and crash detection. MR (magnetoresistive) sensors precise- 
ly positioned in MID packages are used for motion and position 
detection. MID provides 3D-positioning of laser diodes and LED's 
(light emitting diodes) for applications varying from lighting to 
"adaptive cruise control". Tip switches e.g. for steering wheel can be 
realized on 3D-MID structures. They require reduced space and 
allow therefore the integration of more electronic functions. 



1 MID Technology Overview 

The different MID technologies are primarily separated by the manner how to 
structure the metallization. The most popular MID methods are 2-shot-mold- 
ing, LDS (laser direct structuring), LSS (laser subtractive structuring) or hot 
embossing. These technologies are well known and will not be further dis- 
cussed here. Each of these technologies has its advantages and drawbacks for 
specific applications. Relevant criteria of selection are feasibility aspects due to 
the specific limits of each technology, the required design flexibility and the 
appropriate cost to investment relationship. Due to the high volumes and the 
full 3-D capability we usually recommend 2-shot molding as the primary MID 
technology in automotive applications, mainly driven by cost issues. 



2 Advantages of MID Technology for Automotive Applications 

Based on current customer projects and requests we identified the following 
major and unique benefits of MID in automotive applications: 
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Existing mechanical systems in a car can be "upgraded" with electronics (e.g. 
with sensors) by means of MID structures without having to redesign the 
complete system. 

3D-MID structures (e.g. for tip switches) allow the integration of more elec- 
tronics in the limited available space. 

Especially for sensor packaging applications the MID technology has the addi- 
tional advantage of multifunctionality: 

► the package can also serve as substrate for sensor chip, 

► a flow channel can be integrated, 

► pressure port or a tube connection can be realized , 

► RF antenna can be part of a housing, 

► shielding is a possible option, 

► switches are easily implemented and 

► mechanical and electrical interfaces are part of the housing. 



2.1 Design Options 

Some of the available MID technologies offer a high design flexibility because 
high efficient laser processes are used to write the interconnect lines. 
Therefore customer specific solutions are possible without additional tooling 
costs. Not only can the electrical connection lines within one package be 
changed easily, it is also possible to use the laser writing process for marking 
the devices piece or lot wise if this is required. An advantage of this procedure 
is that the traceability is realized at an early state of production. 

For MID components in micro packaging applications high performance ther- 
moplastic materials such as LCP (liquid crystal polymer) are available. They 
offer compatibility with lead-free soldering and the processes for chip mount- 
ing. The outstanding compatibility with harsh environments is another impor- 
tant aspect of the system integration advantage offered by MID devices, 
because peripheral devices must withstand hard conditions outside the central 
electronics boxes. 

To complete the functionality of a peripheral electronic or sensor device it has 
to be connected to the wiring harness in most cases. For actual solutions press- 
in contacts, with additional soldering or conductive gluing are used to achieve 
a reliable connectivity interface to the harness. In the near future it is pre- 
dicted that direct to MID pluggable contacts will be qualified. 
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2.2 The Options of Multifunctionality using MID in Detail 

Mounting chips to a precisely preformed mold body allows to keep narrow tol- 
erances in position and angle with reference to application specific demands. 
Since high accuracy and high precision (precise angles) are already given by 
the injection molded part no additional costs for accurate positioning or trim- 
ming occur (Fig. 1). 




Fig. 1. 3D-MID Package with 2 MR Sensors in 90° tilted position. NCA flip- 
chip assembly. SMD Package, 1,7 mm overall height 

Die bonding, wire bonding, flip-chip and soldering of SMT (surface mount 
technology) devices are possible on MID substrates (Fig. 2). Consequently MID 
packages offer the possibility to integrate the entire system functionality into 
one package. Integration is also the handle to reduce the amount of compo- 
nents. A PCB or Flex-Circuit Board is substituted by the package integrated 
interconnects. 




Fig. 2. NCA flip-chip assembly in detail. Cross-section through the contact 
pads 

Keeping the production efforts nearly at device level, but reaching the func- 
tionality of a system device, offers potentials for miniaturization and ratio. 
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3 Examples of MID Sensor Packages in Automotive Applications 

Based on running customer projects and requests we identified four main auto- 
motive sensor applications where MID based packages are very well suited: 



3.1 Pressure and Flow Sensors for Engine Management, Air Conditioning and 
Crash Detection. 

The 3-dimensional design space available using MID technology allows the 
placement of sensing element and signal conditioning ASIC in optimized posi- 
tions (Fig. 5). Requirement like 

► short lines between sensor and ASIC for signal quality 

► double pressure inlet for differential pressure sensing, 

► additional SMD peripherals inside the housing and 

► stress minimizing sensor mounts (Fig. 4) 

are achievable. 



For a concrete realization a concept with stacked arrangement of the chips and 
wire bond technology was chosen (Fig. 3). 




Fig. 3. Pressure sensor in MID package. View into the open package with 
sensor element mounted 
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Fig. 4. CAD-cross-section through the completed sensor device. Interposer 
layers minimize stress on sensor chip. 




Fig. 5. CAD-cross-section through an alternative design using NCA-flip- 
chip assembly technology 

3.2 MR (Magnetoresistive) Sensors for Motion and Position Detection 

MID allows accurate 3D-positioning of MR chips. Special requirements like the 
nearly coaxial arrangement of two rectangular oriented sensor chips are real- 
izable using MID technology (Fig. 6 & 7). 




Fig. 6. 



Rectangular arrangement of 2 MR angular sensors promising pre- 
cise coaxial alignment [1] 
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Fig. 7. MID substrates for MR positioning and angular sensors [1] 



3.3 Switches on MID 

Tip switches are more and more used to make parts of a car to become intelli- 
gent. The steering wheel is an example where a h uman to electronics inter- 
face is integrated into a mechanical part. 3D-MID structures require reduced 
space and allow a 3-dimensional, handling optimized arrangement of switch- 
es. A snap dome placed on a MID part makes a switch possible nearly every- 
where it is needed. Consequently the integration of more electronic functions 
can be performed with high customer acceptance. 

A switch application using snap domes has been pre-qualified. It was found 
that MID metallization layers fulfill the requirements for industrial applica- 
tion. After more than 4 Mio cycles the switch samples were still working. 
This indicates that the industrial requirement to achieve a guaranteed shelf 
live of 1 Mio. cycles can be fulfilled. 



4 Qualification Status 

HARTING currently qualifies MID technology including chip and SMD assem- 
bly on MID substrates for automotive applications. 

The proof of reliability for a new technology is typically given after a decade 
in field application. HARTING used the established procedures to predict the 
reliability of MID devices. 

Potential failure mechanisms were identified and appropriate accelerated 
aging test were applied to test samples. Today the reliability of the substrate 
materials, the metallization layers and the important interconnections are sue- 
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cessfully proofed. For Laser-Direct-Structuring (LDS) technology in the combi- 
nation with SMD-and NCA-flip-chip-assembly the full range of standard aging 
procedures has been successfully passed. This includes 1000 cycles thermal 
shock -40°C - +125°C and storage in wet heat 85%rH/85°C and dry heat 
125°C. 

To achieve a high design quality for new products, design rules have been 
derived from the results of the qualification procedures. 

Today it is possible to predict a sufficient reliability for selected materials and 
design conditions. 
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Abstract 

AIDE is an integrated project funded by the EC in the 6 th Framework 
Programme. The project, which involves 30 partners including all 
major European vehicle manufacturers, deals with behavioural and 
technical issues related to automotive human-machine interface 
(HMI) design, with a particular focus on HMI integration and adap- 
tation. The project involves tightly integrated empirical research, 
driver behaviour modelling and methodological- as well technologi- 
cal development. This paper provides an overview of the mid-term 
results achieved about half-way through the four-year project. 



1 Introduction 

Every year, about 45 000 people die and 1.5 millions people are injured in traf- 
fic accidents in Europe. It is believed that the great majority of road accidents 
(about 90-95%) involve human error in the causal chain [1]. Recent empirical 
work, such as the US 100 car naturalistic field study, has providing an 
increased understanding of the nature of these errors, pointing to the key role 
of inattention. For example, 93% of the lead vehicle crashes recorded in the 
study involved inattention to the forward roadway as a contributing factor [2]. 
While conventional vehicle safety measures (e.g. seatbelts and airbags) have 
contributed significantly to the reduction of fatalities in the last decades, their 
safety contribution is reaching its limits and currently further improvement is 
difficult to achieve at a reasonable cost. Today, the development of new 
Advanced Driver Assistance Systems (ADAS, e.g. collision avoidance-, lane- 
keeping aid- and vision enhancement systems) offers great potential for fur- 
ther improving road safety, in particular by means of mitigating driver errors. 
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In addition, the number In-vehicle Information Systems (IVIS) increases rapid- 
ly in today's vehicles. This systems has the potential to greatly enhance mobil- 
ity and comfort (e.g. navigation aids & traffic information systems, media play- 
ers, web browsers, etc), but at the same time increase the risk for excessive 
and dangerous levels of inattention to the vehicle control task. Furthermore, 
many IVIS functions are today featured on portable computing systems, often 
referred to as nomadic devices, such as PDAs or advanced mobile phones, which 
are generally not designed for use while driving. In the near future, these 
many of these functions could be expected to be easily downloadable from a 
remote service centre, directly to the vehicle or the nomadic device. 

This proliferation of systems and functions that interact with the driver in one 
way or another leads to a number of challenges, both technical and human fac- 
tors related, for the designer of the future automotive HMIs. These challenges 
include the HMI design for the individual systems as well as he question how 
to integrate a range of different systems into a functioning whole with respect 
to their interaction with the driver. Another challenge concerns how to best 
exploit the technological possibilities of adaptive interfaces in the automotive 
domain. The general objective of the AIDE (Adaptive Integrated Driver-vehicle 
InterfacE) project is to address these challenges by means of tightly integrat- 
ed interdisciplinary work, involving leading human factors as well as technical 
expertise. Specific topics addressed by the project include: 

► The development of a detailed understanding of the behavioural effects 
of interacting with different types of in-vehicle functions 

► The design of human-machine interaction (HMI) strategies for ADAS 
that maximises the safety potential of the systems 

► The design of HMI strategies for IVIS that minimise the added workload 
and distraction to the driving task 

► The development of strategies for integrating multiple in-vehicle func- 
tions into a coherent interface towards the driver 

► The development of technological solutions, e.g. multimodal input-out- 
put devices, driver-vehicle-environment monitoring techniques, and an 
underlying software architecture that supports this 

► The development of solutions for the safe integration of nomadic 
devices into the driving environment 

► The development of valid and cost-efficient methods for in-vehicle HMI 
evaluation with respect to safety and usability. 

A general goal of the project is the realisation and demonstration of the 
Adaptive Integrated Driver-vehicle Interface concept. This entails a unified human- 
machine interface that resolves conflicts and exploits synergies between dif- 
ferent in-vehicle functions. Some key features of the AIDE concept, further 
illustrated in Fig. 1, are: 
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► Multimodal HMI I/O devices shared by different ADAS and IVIS (e.g. 
head-up displays, speech input/output, seats vibrators, haptic input 
devices, directional sound output) 

► A centralised intelligence for resolving conflicts between systems (e.g. by 
means of information prioritisation and scheduling). 

► Seamless integration of nomadic devices into the on-board driver-vehicle 
interface. 

► Adaptivity of the integrated HMI to the current driver state/driving con- 
text. The adaptive interface should also be re-configurable for the dif- 
ferent drivers' characteristics, needs and preferences. This requires 
techniques for real-time monitoring of the state of the driver-vehicle- 
interface system. 




visual . ha P tic J auditory | 




Driver 



Fig. 1 . Illustration of the AIDE Concept 

AIDE is a so-called Integrated Project, funded by the European Commission in 
the Sixth Framework Programme (FP6). It involves 29 partners, including all 
major European vehicle manufacturers, the main suppliers and a range of lead- 
ing research institutes and universities. It is part of the EUCAR Integrated 
Safety Program, which also includes the parallel projects PReVENT, EASIS, 
GST and APROSYS. The project started in March 2004 and will continue for 
four years. The objective of this paper is to provide an overview of the results 
achieved half-way through the project. 

The following chapter briefly describes the general structure of the project. 
Chapter 3 then provides a review of the main results obtained in the four sub- 
projects so far, and some hints on the next steps to be taken in the second half 
of the project. Finally, Chapter 4 provides a general discussion on the results 
obtained so far in relation to the general project objectives. 
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2 Project Structure 

As mentioned above, AIDE is a so-called Integrated Project. This a new instru- 
ment introduced by the EC in FP6, where a set of relatively large sub-projects 
(SPs) are integrated towards a common goal. AIDE consists of four such sub- 
projects: 

► Sub-project 1: Behavioural effects and driver-vehicle environment modelling: 
The main goal of this sub-project is to develop a generic model and com- 
puter simulation of driver-vehicle-environment interaction, focusing on 
drivers' interaction with ADAS and IVIS. This will be based on empiri- 
cal results from experiments on short- and long-term behavioural 
effects of different driver support functions. 

► Sub-project 2: Evaluation and assessment methodology: The main goal of this 
sub-project is to develop a generic valid and cost-efficient methodology 
for safety- and usability evaluation of in-vehicle functions and to apply 
this method in the final evaluation of the AIDE prototype vehicles 
developed in SP3. 

► Sub-project 3: Design and Development of an Adaptive Integrated Interface: 
The general objective of this sub-project is to design the AIDE system 
and develop the AIDE demonstrator vehicles. This includes the require- 
ment specification, development of the system architecture, HMI 
design and the definition of HMI strategies, solutions for nomadic 
device integration, development of DVE monitoring modules and 
demonstrator vehicle implementation. 

► Sub-project 4: Horizontal issues: While most of the research and techno- 
logical development (RTD) is performed in SP1-3, SP4 contains a num- 
ber of horizontal activities such as project administrative and technical 
management, dissemination and exploitation, development of HMI 
guidelines and standards and internal review and assessment. 



Sub-project 4 

Horizontal aclivilies 



Sub- project 1 

Behavioural Effeds 
and Driver-Vehicle- 
Environmenl 
Modelling 



Sub-project 2 

Evaluation and 
Assessment 
Methods 



Sub-projectS 



Development of an 
Adaptive Integrated 
Drivsr-vehicte 
Interface 



Fig. 2. General structure of the AIDE project 
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The main expected result from the project is the demonstration of the AIDE 
concept in three demonstrator vehicles, one city car, one luxury car and one 
heavy truck. The evaluation methodologies developed in SP2 will be used both 
during the iterative HMI design and development and in the final evaluation 
of the demonstrators. Both the SP3 HMI design- and the SP2 methodology 
development is supported by empirical and analytical results from SP1 on the 
short- and long-term behavioural effects of in-vehicle systems. Thus, in addi- 
tion to supporting the overall goal of developing and demonstrating the AIDE 
concept, each sub-project will deliver important results on their own, in par- 
ticular the SP1 DVE model/simulation and the SP2 validated evaluation 
methodology. 



3 Review of Mid-Term Results 

After about 2 years of work, the project is in the middle of its most intense 
development and experimental phase. The present chapter provides an 
overview of the results achieved to date. Naturally, due to the limited space, 
only general descriptions can be provided here. However, references are pro- 
vided to sources where more detailed information can be obtained, in particu- 
lar the AIDE deliverables most of which are available on www.aide-eu.org 
(where also summaries of the confidential deliverables can be found). 



3.1 Sub Project 1 : Behavioural Effects of Driving Support Systems and Driver- 
Vehicle-Environment Modelling 

The main goal of SP1 is to develop a detailed understanding about the mecha- 
nisms that govern the interaction between the driver, the vehicle (in particu- 
lar with IVIS and ADAS) and the environment, to be embodied in testable 
models as well as a computer simulations. The model development is support- 
ed by empirical studies on behavioural effects of different types of support sys- 
tems. Below, the main results to achieved so far are reviewed. 



3.1.1 Driver-Vehicle-Environment Modeling and Simulation 

The main expected result from this activity is a testable model of the DVE, 
implemented in a computer simulation, which can be used to investigate 
potential behavioural effects early in the design process. The results of this 
work is also an important input to the DVE monitoring module development in 
SP3 (see section 3.3.6). 
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The development of the AIDE DVE model is an iterative, incremental proce- 
dure which is based on data collected in the AIDE empirical studies on behav- 
ioural effects of driver support systems (see next section) as well existing 
results in the literature. The basic requirements on the model and a prelimi- 
nary model structure were outlined in Cacciabue et al., [3]. The modelling is 
based on the general SHELL architecture [4], which is a suitable framework for 
representing interactions between humans and other elements in the working 
environment such as support systems and other humans. A major focus so far 
has been on the identification of the key parameters of the model. At the cur- 
rent stage of development, the following parameters have been identified: (1) 
Attitudes/personality, (2) experience/competence, (3) task demand, (4) driver 
state, (5) situation awareness and (6) driver intention/goals. For each parame- 
ter, the correlation to measurable variables and other parameters are investi- 
gated. The results are documented in Cacciabue et al. [5]. (It could be noted 
that these parameters correspond roughly to the output vector of the DVE 
monitoring modules developed in SP3 - see below) 

Moreover, a first high-level specification of the simulation software architec- 
ture has been developed. The DVE simulation architecture is intended as a 
generic tool where different types of driver, environment and vehicle models 
could be implemented and tested. For a further description of the initial DVE 
simulation specification, see Carusi [6]. 



3.1.2 Empirical Results from the Learning Phase Experiments 

Behavioural effects of in-vehicle functions could be divided into direct and indi- 
rect effects. The former refer to behavioural changes that are intended by the 
designers of the functions, such as reduced speed resulting from the introduc- 
tion of a speed alert function. By contrast, indirect effects are non-intended 
side-effects resulting from the use of a system. Moreover, when studying 
behavioural effects it is important to distinguish between short-term effects that 
occur during when the driver is learning to use the system, and long-term 
effects , which refers to behavioural changes over a longer period of using the 
system. For a literature review of known behavioural effects of in-vehicle func- 
tions, performed within AIDE, see Saad et al. [7]. 

In AIDE SP1, two sets of experiments were defined, one focusing on the learn- 
ing phase, with a particular focus on integration and adaptivity, and the other 
focusing on long term effects. To date, the learning experiments have been 
completed. These studies included three coordinated simulator experiments 
investigating adaptivity of a forward collision warning (FCW) function to (1) 
driver distraction, (2) road friction and (3) individual driver characteristics. 
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Moreover, a field experiment addressed learning effects of a standard 
Adaptive Cruise Control (ACC) system and another field experiment looked at 
learning effects of speed limiter and cruise control functions with a common 
interface towards the driver. Finally yet another field experiment looked at 
individual and combined effects of forward collision warning (FCW) and lane 
departure warning (LDW) systems. 

An interesting finding from the LDW-FCW experiment, conducted by the 
Hellenic Institute of Transport, was that subjects tended to properly adapt 
their behaviour in the intended way when exposed to one of the two systems, 
while these positive effects did not occur to the same degree when the two 
systems were active simultaneously. This strongly indicates the behavioural 
effect of combining two warning systems is not simply additive (i.e. sum of the 
effects of the systems in isolation) and points to need for integrated warning 
strategies. 

A common finding from the learning phase studies was that individual driver 
characteristics, such as personality traits, is a key factor influencing behav- 
ioural effects and acceptance of driver support functions. For example, in one 
of the studies on adaptive FCW, conducted by Leeds University, the warning 
criterion was adapted to the individual driver's reaction time, measured in the 
simulator prior to the main trials. Thus, in the adaptive system condition, driv- 
ers with fast reaction times received the warning later than drivers with slow 
reaction times. Moreover, personality traits such as sensation seeking were 
measured as well as the driver's preferred headway. In general, the results 
showed that both the non-adaptive and adaptive FCW reduced the reaction 
time to critical events indicating a clear safety benefit of the FCW function. 
Moreover, for non-aggressive drivers (long-followers and drivers with low-sen- 
sation seeking scores), there was no difference in acceptance ratings between 
the adaptive and non-adaptive systems. However, aggressive drivers (short fol- 
lowers with high sensation seeking scores), rated the adaptive system signifi- 
cantly more useful and satisfactory than the non-adaptive system. Since 
acceptance of a driver support system is the key to market- and safety impact, 
this result strongly indicates the value of adapting warning thresholds to indi- 
vidual driver characteristics. Besides feeding the DVE model development, 
these results thus provide important input to the development of adaptive 
warning strategies in AIDE SP3, especially with respect to the development of 
the Driver Characteristics module, with the purpose to record and monitor 
both static and dynamic driver characteristics (see section 3.3.6). The results 
from the learning phase experiments will be reported in Deliverable 1.2.3 
(which will be finalised after the submission of this paper). 
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3.1.3 Next Steps, S PI 

The main next step of the empirical work in SP1 is the long-term experiments, 
which will be finalised by August 2006. These experiments will include: (1) 
The continuation of the FCW-LDW study mentioned above, beyond the learn- 
ing phase, (2) a larger-scale study of the speed-limiter/cruise control system 
mentioned above, (3) a long-term study of a product LDW system and (4) a 
study on long-term effects of Intelligent Speed Adaptation. 

The DVE modelling work will continue with refinement of the model based on 
the empirical results reviewed above, the implementation into a running sim- 
ulation and the final validation of the model/simulation. 



3.2 Sub-Project 2: Evaluation and Assessment Methodology 

The main objective of AIDE Sub-project 2 is the development of a generic 
methodology for evaluation of automotive HMI solutions with respect to safe- 
ty and usability. In the final stage of the project, the developed methodology 
will be used in the final human factors evaluation of the AIDE prototypes. 
Initial work focused on extensive reviews of existing safety and usability meth- 
ods and tools (see Cherri, Nodari and Toffetti, [8] and Johansson et al., [9]). 
Subsequent work has focused on parallel development of the individual build- 
ings blocks of the evaluation methodology, in particular evaluation scenarios, 
workload/distraction assessment techniques, and methods for relating behav- 
iour/performance measures into measures of risk. The results from the latter 
activities are reviewed below. 



3.2.1 Evaluation Scenarios 

The scenario specification is a central topic in the development of a generic 
HMI evaluation methodology. The scenario specifies the context in which the 
evaluation takes place, including the test vehicle, the road infrastructure, the 
traffic conditions and the environmental conditions. The work in AIDE SP2 on 
evaluation scenario definition has so far focused on collecting relevant infor- 
mation from within and outside the project, and identifying the key building 
blocks for AIDE evaluation scenario definition. An important first step was to 
review, and define a taxonomy for, IVIS/ADAS systems (Floudas et al., [10]). 
Moreover, categorisation schemes for driving simulators, subject groups etc. 
were defined, and the AIDE design scenarios, defined in SP3, were reviewed 
and related to the evaluation methodology development. Also, key results 
from the HASTE and COMUNICAR FP5 EU-funded projects were reviewed. 
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Based on this, the key building blocks for the AIDE evaluation scenario defini- 
tion were identified. These building blocks were grouped into three main cat- 
egories: (1) Road type and conditions (e.g. highway rural road, visibility con- 
ditions), (2) Traffic type and actors (e.g. oncoming traffic, pedestrians etc.) and 
(3) Tasks and goals (e.g. car following, overtaking, distraction task etc.). The 
building blocks were then mapped onto the different IVIS/ADAS categories, 
defined in [10], by means of a matrix representing experts' judgements on the 
relevance of the different scenario building blocks for the evaluation of each 
type of system. A similar matrix was also constructed for the simulator/test- 
vehicle categories. These results are described in Rimini-Doring et al. [11]. 



3.2.2 Workload and Distraction Assessment Methods and Tools 

Substantial work in SP2 has also been performed on methods and tools for 
offline driver workload and distraction assessment. Key results from this work 
include: 

The Visual Demand Measurement Tool: A software tool for eye and head 
movement analysis, known as the VDM (Visual Demand Measurement) Tool, 
was developed. This tool can be used for automated analysis of eye- and head 
movement data from different eye-tracking systems and contains the basic 
signal processing algorithms for noise filtering and glance segmentation, a 
comprehensive set of functions for data management and visualisation as well 
as an easy-to-use graphical user interface. 

Secondary task-based methods: This involved the development of improved 
methods for secondary task-based measurement of driver workload, resulting 
in new versions of the Peripheral Detection Task (PDT) method (see e.g. [12] for 
a description of the original PDT method. While the standard PDT is based on 
the ability to detect visual stimuli presented in the periphery, the present work 
investigated a range of different detection task methodologies, including ver- 
sions where the stimuli were presented in tactile and auditory modalities. In a 
set of parallel experiments, it was found that the placement of the visual stim- 
uli did not have a significant impact on the sensitivity to secondary task work- 
load. Moreover, the sensitivity was relatively consistent even across modali- 
ties. Based on these results, specifications for new tactile and visual detection 
task methodologies were derived, where tactile was recommended as the 
default modality. The results from this work are further reported in Merat et 
al., [13]. 

Driving performance metrics: Part of this work addressed traditional vehicle 
control metrics such as speed, headway, lateral control performance and steer- 
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ing wheel movements, while another sub-task focused on a new influential 
methodology known as the Lane Change Test (LCT) [14]. The work on tradi- 
tional metrics was largely based on results and data from the HASTE FP5 EU- 
funded project, and addressed a number of issues that were left open from this 
project. Based on some re-analysis of the HASTE data, a subset of vehicle con- 
trol metrics, considered suitable for the AIDE Evaluation Methodology, were 
selected and specified in detail. One specific focus in this task was the devel- 
opment of improved steering wheel metrics for workload and distraction 
assessment. A modified steering reversal rate metric, with different parame- 
ter settings for visual and cognitive load, was defined. A set of experiments 
were also conducted on the Lane Change Test. For example, one of these stud- 
ies compared LCT results from three different simulator setups: a PC-based 
low-fidelity set-up, a driving simulator with a plasma screen and a driving sim- 
ulator with full projection. It was found that simulator set-up did have a sig- 
nificant effect on LCT results, where increasing simulator grade led to decreas- 
ing LCT scores (lane deviation values). However, no difference was found with 
respect to the IVIS tasks included, and thus it could not be determined 
whether the scores scale linearly between the different set-ups. The results 
from this work are documented in [15]. 

Subjective workload scales: This work focused on an experimental compari- 
son of a number of a number of existing subjective workload scales. The scales 
investigated were (1) PSA-TLX (PSA-Task Load indeX), (2) the DALI (Driving 
Activity Load Index) questionnaire and (3) the BMDMW (Behavioural Markers 
of Driver Mental Workload). The scales were evaluated in 3 parallel experi- 
ments: one in simulator and two in real-world conditions. It was concluded that 
all three methods were sensitive to workload but that they are applicable to 
different types of evaluation situations and purposes, and thus could be con- 
sidered as roughly complementary. Recommendations for the application of 
each method were provided. See Chin et al. [16] for further details. 

In the next step, the methods and tools resulting from all these sub-tasks will 
be empirically compared in a single study and the results fed into the general 
development of the AIDE evaluation methodology. 



3.2.3 The Relation between Behaviour and Risk 

One of the most important issues in the entire AIDE project, but at the same 
time one of the most difficult, is how to estimate the (positive or negative) safe- 
ty effects of the changes in driver behaviour and state induced by IVIS, ADAS 
and different integrated/adaptive HMI solutions. This issue is specifically 
addressed as part of the SP2 work. While existing approaches, such as [2], has 
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focused on direct accident analysis, the work in SP2 has investigated a range 
of alternative approaches that should be considered as complementary to acci- 
dent analysis studies. Specifically the work has included the identification of 
basic driver behaviour-risk relations (e.g., between speed and accident risk), 
based on the literature as well as on new material generated by a re-analysis 
of data from the HASTE Project. Moreover, an empirical simulator study inves- 
tigated the relations between situational demand, driving performance and 
accident risk. Finally, the potential of micro-simulation as a tool for getting to 
the desired functions was investigated. The results are documented in Jamson 
et al. [17]. Further work has looked into how the combined effects of behaviour 
parameters (e.g. speed) and driver state parameters (in particular workload) 
could be measured, resulting in a pragmatic and quantitative proposed 
method for mapping behaviour and driver state to risk. See Janssen, Brouwer 
and Huang [18] for details. 



3.2.4 Next Steps, SP2 

In the next phase of the project, the results from the parallel activities 
reviewed in sections 2.2.1, 2.2.2 and 2.2.3 above will be integrated into a 
generic evaluation methodology, which will be empirically validated in a num- 
ber of emprical studies. Since the main part of the SP2 development work is 
focused on IVIS-related evaluation, further input on ADAS-related evaluation 
methods will also be incorporated from SP1 and the PReVENT sub-project 
Response 3. Based, on these inputs, the final AIDE evaluation methodology will 
be specified and applied in the final human factors evaluation of the AIDE pro- 
totype vehicles developed in SP3. 



3.3 Sub-Project 3: Design and Development of an Adaptive Integrated 
Interface 

SP3 is responsible for the actual design and development of the AIDE system. 
At the current stage, the specification phase has been concluded and the sub- 
project is now in the middle of its main design and development phase. The 
design of the adaptive integrated HMI is performed in an iterative, incremen- 
tal, way, according to the standard User Centred Design (UCD) procedure. This 
implies that users are involved from the beginning of the design process, e.g. 
in evaluating early versions of the systems implemented in virtual prototypes. 
The final expected result from SP3 is the three prototype vehicles that will 
demonstrate the AIDE concept. The SP3 design process is supported by empir- 
ical results on behavioural effects of ADAS/IVIS from SP1 as well as evaluation 
methods and tools developed in SP2. 
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3.3.1 The AIDE use Cases, Design Scenarios and Requirements 

The first step in the design and development of the AIDE system in SP3 was 
to define the scenarios and use cases that the AIDE system should address. 
This was partly based on a user needs analysis documented in Amditis et al 
[12]. As described in the introduction, the AIDE system is intended to work as 
an overall manager of the in-vehicle HMI. Thus the HMI management func- 
tions can be considered as meta functions, responsible for managing individ- 
ual ADAS and IVIS with respect to their interaction with the driver. An exam- 
ple of such a meta function could be to block non-critical information (e.g. 
phone calls) in demanding driving situations. Since each such meta function 
generally covers a large number of specific cases (e.g. a phone call/traffic infor- 
mation/email is blocked in an intersection/during overtaking/in a sharp curve), 
a formal method for defining generalised use cases was needed. Since such a 
method did not exist, it had to be invented in the project. The result was the 
definition of a set of generalised problem scenarios, known as the AIDE design 
scenarios. 

The AIDE design scenarios were specified in terms of a formalism using two 
main types of parameters: (1) application actions and (2) Drive r-Vehicle- 
Environment (DVE) conditions. The actions represent an event initiated by the 
user or a system while the DVE conditions represent the momentary state of 
the Driver-Vehicle-Environment system. In order to obtain a generalised 
description, parameterisation schemes were developed for both the actions 
and DVE conditions. The actions were described in terms of eight parameters: 
Initiator {User, System}, Duration {Transient, Sustained}, Safety Criticality 
{None, Low, High}, Time Criticality {None, Low, High}, Real Time, Mandatory 
{Yes, No}, Driving Relevance {Yes, No} and Preference {Yes, No}. The DVE 
conditions were represented by five main parameters: Driving demand. 
Distraction, Driver Impairment, Driver Intent and Traffic and Environmental 
Risk. The actions were then classified into a set of priority classes, defined in 
terms of the action parameters. Based on this categorisation scheme, a set of 
general AIDE design scenarios were defined. The action and DVE condition 
parameterisation and classification schemes were also the starting points for 
the definition of the logic of the AIDE information management functions, 
implemented mainly by the Interaction and Communication Assistant (ICA, 
see section 3.3.3). In addition to the generalised AIDE design scenarios, a num- 
ber or "normal" use cases were defined for nomad device integration and some 
adaptive HMI functions. The AIDE user needs, use cases and design scenarios 
are documented in Amditis et al. [12]. 

Based on the user needs analysis and the defined scenarios and use cases, a set 
of basic requirements for the AIDE system were derived. These included func- 
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tional as well as non-functional requirements, where the former refers to the 
required functionality of the system (e.g. the AIDE meta-functions, the sup- 
ported I/O devices, supported nomadic device integration functionality etc.) 
and the latter refers to technological requirements such as scalability, modu- 
larity etc. In addition to these general requirements, a set of specific imple- 
mentation requirements for the three demonstrator vehicles were defined. 
The AIDE requirements are documented in Kussman et al [20]. 



3.3.2 The AIDE Software Architecture 

Based on the design scenarios and requirements, a general software architec- 
ture for the AIDE system was developed. This work focused on high-level, log- 
ical, architecture descriptions, which are largely independent of the underly- 
ing electronics HW and SW architecture (e.g. bus systems, gateways, middle- 
ware etc.). However, it is obviously important to identify the requirements that 
the AIDE logical architecture put on the general electronics architecture. This 
topic is currently addressed in cooperation with the parallel EASIS EU-funded 
project. The general role of the AIDE-specific architecture work was to define 
the main logical components, their roles and mutual relations. 

The key components in the AIDE system are the Applications, the I/O devices, 
the Interaction and Communication Assistant (ICA) and the Driver vehicle 
environment (DVE) monitoring modules. The basic principles behind the archi- 
tecture are illustrated in Fig. 3. Applications communicate with ICA by means 
of four types of messages. The Application Request Vector (ARV) is used to ask 
ICA for permission to execute an action (e.g. presenting output). The applica- 
tion request consists of a vector with action parameters, similar to the vector 
used in the AIDE design scenarios (described above). Based on the application 
request, the ICA assigns the action to a priority class. Based on this priority, 
the presence of other possible running actions and the current DVE state, the 
ICA decides which HMI strategy to apply. A Reply Vector (RV) is then sent back 
to the application which acts accordingly (e.g. by modifying postponing or ter- 
minating the action). If an application has finished an action it has to inform 
the ICA, using the Channel Status Vector ( CSV). Moreover, if a requested output 
becomes invalid during a postpone phase, the application informs ICA about 
the new status using the Request No More Valid Vector (RNV). 

An important property of this architecture solution is that, due to the fact that 
the ICA only operates on the action parameters (and not directly on the action 
content), applications can be added and removed without the need to change 
the ICA software. This greatly enhances flexibility and modularity. Moreover, 
since the priority assignment is done in the ICA, the vehicle manufacturer has 
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a certain control over the prioritisation strategy. However, it should be noted 
that this requires standardised protocols for the request and reply vectors. 
Another important issue concerns the role of the applications. While true HMI 
integration inevitably will influence the individual applications, it is desirable 
to keep this influence as small as possible. In the AIDE architecture, the main 
addition to the applications is the software needed to implement the commu- 
nication with the ICA, a sub-component known as the AIDE interface adapter. 
By means of structuring the application according to the model-view-control 
pattern [21, illustrated in Fig. 3], a logical separation between the basic appli- 
cation SW and the AIDE-specific additional SW can be obtained, which great- 
ly facilitates the SW development process. 

The role of the Driver- Vehicle-Environment (DVE) monitoring component is to 
feed both the ICA and the applications with a momentary representation of 
the DVE state, which is represented by a DVE vector similar to the represen- 
tation used in the AIDE design scenario descriptions. The DVE vector also 
includes a representation of static and dynamic driver characteristics, provid- 
ed by the Driver Characteristics module. The DVE modules are further 
described below in section 3.3.6. See Kussman et al. [22] for a more detailed 
description of the AIDE architecture. 




Fig. 3. Basic principles of the AIDE architecture 
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3.3.3 General HMI Strategies and the Interaction and Communication Assistant 

The AIDE HMI strategies include the basic HMI design and configuration (i.e. 
the default mapping of actions, formats and input/output channels) as well as 
the dynamic information management taking into account the DVE state inter- 
dependencies (e.g. conflicts) between applications. The component with the 
main responsibility for realising the HMI strategies is the Interaction and 
Communication Assistant (ICA). The ICA can be viewed as the "brain" of the 
AIDE system and is responsible for the centralised management of the inte- 
grated HMI by implementing the AIDE meta functions mentioned above. This 
is based on real-time context awareness provided by the DVE monitoring mod- 
ules. Specifically the ICA implements the logic that determines if, when and 
how information should be presented to the driver. Thus, the ICA handles con- 
flicts between applications (e.g. by means of prioritisation or change of output 
channel) as well as conflicts between individual applications and the driving 
task (e.g. by means of delaying non-critical information in demanding driving 
situations). As described above, a basic principle behind the AIDE solution for 
HMI integration is that each application always requests permission from the 
ICA before executing an action, using the application request vector (ARV) for- 
mat described above. The application request is handled by the ICA logic s 
series of steps, implemented in four distinct sub-modules: 

► The Priority Manager : Receives the application requests and assigns a 
priority to the requested action based on the information in the ARV. 

► The Filter: Decides whether the action can be presented given the cur- 
rent DVE conditions (as provided by the DVE monitoring modules). 

► Modality Selector: Decides the whether modality (e.g. intensity, format 
etc.) of the message should be adapted according to the current DVE 
conditions. If not the request is put in a waiting queue until the DVE 
condition changes. 

► The Channel Selector: Decides the output channel for the action. For each 
action, a primary (preferred) and optionally a secondary, output chan- 
nel is specified in the ARV. The Channel module first checks whether 
the primary channel is free and, if not, the availability of the secondary 
channel is checked (given that the HMI strategies permit presentation 
in the secondary channel). If all channels are blocked, the request is put 
in a waiting queue. 

When the HMI strategy has been decided, the ICA informs the application 
with a Reply Vector (RV) in a specified format. The internal ICA architecture 
and its relation to the other main AIDE components are further illustrated in 
Fig. 4. 
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Fig. 4. Basic Principles of the ICA Logic 

In order to perform early testing of the ICA logic, a special-purpose simulation 
tool has been developed. The main objective of this tool is to visualise and test 
the internal ICA logic and its interaction with other components. For more 
details on the ICA development, see Deregibus et al. [23]. 

In addition, specific work is devoted to the design of integrated and adaptive 
ADAS HMI strategies, in close collaboration with the IN SAFES sub-project in 
the PReVENT IP (www.prevent-ip.org). It should be pointed out that, in the 
AIDE architecture, the ICA is not responsible for handling conflicts between 
strongly safety- and time critical warnings. The main reason for this is that 
this particular type of conflict resolution is difficult to separate from the appli- 
cation's own warning strategy. Thus, assigning this responsibility to the ICA 
would blur the responsibility between the ICA and the application which 
would create a dependency between the application and the ICA. Such a 
dependency would violate the basic requirements of modularity and scalabili- 
ty stated above. Thus, the adopted solution was to assign the responsibility for 
time-critical ADAS conflict resolution to a specific ADAS HMI manager (to be 
developed in IN SAFES). However, the ADAS HMI manager always needs to an 
ARV request to the ICA, and await a reply, in order to avoid potential conflicts 
with other applications (e.g. the case when a sound warning is initiated while 
the radio plays loud music). Thus, the ADAS HMI manager has the same logi- 
cal role in the AIDE architecture as a "normal" application. 
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3.3.4 HMI Input/Output Development and the AIDE Virtual Prototypes 

While the AIDE solution should support the whole spectrum of input/output 
(I/O) HMI devices, the actual I/O device development in the project focuses on 
a smaller set of devices including speech input/output, multifunctional haptic 
input devices, head-up displays, configurable displays and tactile seats. While 
some software and hardware development of individual I/O devices is per- 
formed in the project, the main focus is on developing solutions for integrating 
multiple I/O devices into a functioning whole. 

At the current stage of development, the first versions of the AIDE Virtual 
Prototypes (VPs) are being finalised. These represent first implementations of 
the AIDE system in driving simulator environments, which allow for early user 
testing of different HMI solutions. For the first user tests, two Virtual 
Prototypes have been developed, one car- and one truck version (the car VP 
will evaluate both the Seat and Fiat HMI solutions). The very first version of 
the Truck VP is illustrated in Fig. 5 below. The Virtual Prototypes will contain 
most of the envisioned AIDE HMI functionality, including simulated applica- 
tions, dialogue structures, graphical layouts, I/O devices, and some of the AIDE 
meta functions. However, in these initial tests, some of the functionality will 
not be implemented in real-time but rather simulated by means of Wizard-of- 
Oz techniques. 




Fig. 5. The first version of the AIDE truck virtual prototype: The graphics 
for the configurable instrument cluster (top left), the simulated 
infotainment application (top right) and the truck mock-up with 
some of the prototype HMI I/O devices (bottom) 
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3.3.5 Nomadic Device Integration 

Another important topic addressed in the AIDE project is the integration of 
nomadic devices (ND), i.e. mobile devices such as Smartphones and PDAs. 
Work in AIDE on this topic involves both the technical development of possi- 
ble solutions for ND integration as well as the establishment of the Nomadic 
Device Forum. Technical results to date include the identification of the key 
use cases and basic architectural requirements [19-21]. Moreover, a potential 
integration solution, based on a Bluetooth protocol developed by Motorola, has 
been identified. This solution will be implemented and demonstrated during 
the second half of the project. 
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Fig. 6. The architecture of the Driver-Vehicle-Environment modules in 
AIDE 

The Nomadic Device Forum is a consortium of vehicle manufactures, electron- 
ic device manufacturers and telecom industries, public authorities and other 
stakeholders with an interest in integration of ND into vehicles. The forum is 
being run by ERTICO and during its first year of operation, several workshops 
have been held. Topics addressed by the forum include both technical- (e.g. 
how can ND integration be achieved and which are the most feasible protocols) 
and strategical issues (e.g. which are the key business models for the different 
stakeholders). The forum can be accessed at: http://ndf.itsnomadic.net/ 
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3.3.6 Real-Time DVE Monitoring 

As described above, the purpose of the DVE monitoring is to provide a real- 
time assessment of the Driver- Vehicle-Environment state that can be used by 
the ICA, or directly by the applications, to enable the adaptive HMI function- 
ality defined by the AIDE design scenarios and use cases. The general archi- 
tecture of the DVE modules is illustrated in Fig. 6. 

As illustrated in Fig. 5, all DVE modules have access to a shared sensor array. 
This includes driver sensors (e.g. eye-/head-/eyelid tracking), vehicle sensors 
(e.g. steering wheel angle sensor, pedal sensors, gyros etc.) and environment 
sensors (e.g. radar, laser, video). Moreover, a digital map database plus global 
position data is used. Other input data includes static driver profile data 
entered by e.g. a smart card. 

Each of the DVE monitoring modules is responsible for a specific dimension of 
the DVE space. The main role of the respective modules are: 

Driver Availability Estimation (DAE) module: Computes an estimation of the 
availability of the driver to receive information, as determined by the demand- 
of the primary driving task. This is based on vehicle sensor.- as well as digital 
map data. 

Driver state degradation (DSD) module: Computes an estimation of the cur- 
rent drowsiness state, based on a combination of physiological and behaviour- 
al sensor data. 

Cockpit activity assessment module (CAAM): Computes an online estima- 
tion of the driver's secondary task activity and (visual and cognitive) distrac- 
tion level and. The main input is eye-/head tracking data, although other 
information sources such as lane tracking data and in-vehicle button presses 
are used as well. 

Traffic and environment risk assessment (TERA) module: Computes esti- 
mates of traffic and environment risk mainly based on environment sensing, 
but also digital map data. This also includes estimation of drivers' intent to per- 
form driving manoeuvres, partly based on input from the CAAM module. 

Driver Characteristics (DC) module: Maintains a static record of driver char- 
acteristics and preference, based on driver identification (e.g. by means of 
smart card) but also computes a dynamic profile of driver characteristics (e.g. 
reaction time, preferred following distance etc.) computed from sensor data. 
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At the current state of development, first versions of the software for the mod- 
ules have been released [26]. 



3.3.7 Next Steps, SP3 

During the second half of the project, the development of the AIDE compo- 
nents will continue with iterative design and evaluation of virtual prototypes. 
In this process, AIDE simulations and prototypes will be developed and tested 
by a range of vehicle manufacturers, such as Seat, Fiat, Volvo, Ford, Renault 
and Peugeot-Citroen. In the final step, these components will then be inte- 
grated in the three main AIDE demonstrator vehicles, a city car developed by 
Seat, a luxury car from Fiat and a heavy truck from Volvo. When the demon- 
strator vehicles have been technically verified, they will be handed over to SP2 
for human-factors evaluation. In parallel, the nomadic device forum will con- 
tinue its activities. 



3.4 Sub-Project 4: Horizontal Activities 

While most of the work in SP4 is related to the day-to-day management of the 
project, some important technical work has been performed as well. In partic- 
ular, this SP includes work on development of guidelines and proposals for new 
standards based on input from SP1-3. So far, this work has mainly included a 
review of existing guidelines and standards [27], while the actual development 
will take place in the second half of the project. This work has also supported 
the update of the European Statement of Principles on HMI, conducted during 
2005. 

Another result from SP4 is the development of a general conceptual frame- 
work, intended to be used as a common language to conceptualise key issues 
related to HMI and driver behaviour. The framework is based on the 
COCOM/ECOM model [28] and is further described in [29]. A short description 
of the framework and its intended application in the AIDE project can be found 
in [30] the AIDE Interaction Plan [29]. This deliverable also describes the plan 
for interaction between the AIDE SPs and with other related projects, such as 
PReVENT, EASIS, GST and HUMANIST. The 
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4 Discussion: Towards the Future Automotive HMI 

This paper has provided an overview of the mid-term results from the AIDE 
project. This section provides a concluding discussion on these results in rela- 
tion to the key objectives of the project, and points out some key issues that 
need to be further addressed in the second half of the project. 



4.1 HMI Design for Maximising Safety Effects 

The design of warning/intervention strategies is critical for the ability of an 
ADAS to increase safety by changing driver behaviour in the desired way. If 
the warning is not properly designed, the driver may ignore it or shut the sys- 
tem off altogether. An important point is that it is clearly not meaningful to 
separate the warning strategies from their application. Thus, the ongoing 
work on warning strategies in AIDE always needs to be tightly integrated with 
the application development in PReVENT. 

A particular issue of great interest for AIDE is the possibility of adaptive warn- 
ings. Warning timing and/or intensity could be adapted to a number of situa- 
tional factors, such as individual driving style, driver distraction, drowsiness 
and road condition. While this is clearly technologically feasible today (Toyota 
has announced that they will put a distraction-adaptive FCW system on the 
market early 2006), there are many tricky HMI-related issues still left open. 
For example, since the driver is also a (very advanced) adaptive system, it is 
likely that he/she will compensate for system adaptations that are not in line 
with his/her motives. Thus, the behavioural effects of adaptive warnings (e.g. 
in terms of brake reaction time) may not always be easy to predict. However, 
the SP1 results showed that adaptation of warning criteria to the individual 
driving style could enhance acceptance, especially for aggressive drivers. 
Another issue to consider is that system adaptivity also reduces predictability, 
which makes it more difficult for the driver to create a proper mental model 
of the system behaviour. Thus, while some types of adaptive functions may be 
useful, others may not. The potential of adaptive warnings will be further 
investigated in the virtual prototyping and testing in SP3, based on the empir- 
ical results from SP1. The prediction of the mutual adaptations between the 
driver and adaptive warning systems is also a key application for the DVE 
model/simulation. 
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4.2 HMI Design for Minimising Safety reducing Effects of in- Vehicle Systems 

A further important aim of the AIDE project is to develop HMI solutions that 
minimise the negative effects of individual in-vehicle systems in terms of dis- 
traction and workload. A first step towards this goal is the development of 
valid and reliable methods for assessing distraction and workload and their 
effects on driving performance. The work in SP2 has focused on comparing 
and refining existing methods and resolving a number of open issues from pre- 
vious projects such as HASTE, RoadSense and ADAM. However, a further key 
issue concerns how distraction, workload distraction and driving performance 
measures could be related to actual safety. Work in SP2 has investigated a 
range of approaches to address this difficult problem, as a complement to 
direct accident analysis [17-18]. In the next step, these approaches will, 
together with the work on evaluation scenarios [11], be integrated into the 
generic AIDE evaluation methodology. By applying this methodology, it should 
be possible to compare different HMI design alternatives with respect to their 
(positive and negative) effects on road safety. 

Another approach for minimising excessive workload and distraction is by 
means of active workload management, e.g. re-scheduling of non-critical infor- 
mation in demanding driving situations. In AIDE, this functionality is imple- 
mented by the ICA, based on input from the DVE monitoring modules (in par- 
ticular DAE and TER A). An important issue concerns the extent to which these 
types of functions are accepted by different user groups. An important issue 
for SP2 to consider concerns how to evaluate these types of meta-functions 
with respect to their expected safety enhancements. 



4.3 HMI Integration 

While the discussion so far has focused mainly on HMI issues related to indi- 
vidual systems, perhaps the most important issue addressed by the AIDE sys- 
tem is HMI integration. This is a problem that has to be solved if the full poten- 
tial of driver support technologies, in terms of safety and mobility, should be 
realised. Perhaps the most straightforward reason for this is the packing prob- 
lem - if every application had its own dedicated HMI, the HMI devices would 
simply not fit into the vehicle cockpit. However, there are also a number of 
human factors-related reasons for why HMI integration is needed. As the 
results from the SP1 learning studies show, the individual effects of two sys- 
tems may not add up to their effects in combination. For example, if two warn- 
ing systems are added on top of each without any further integration, the 
number of warnings presented to the driver will be doubled, with the possible 
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result that the driver's "acceptance threshold" is reached and the warnings 
ignored (this is one likely explanation for the results in the SP1 study). 

HMI integration can be achieved in different ways. The most basic approach 
(which should always come first) is to take into account interdependencies 
between functions from the early stages of the design process. This also 
includes the design of the basic HMI configuration and the definition of a prop- 
er mapping between functions and I/O devices. A key human factors chal- 
lenge here is to enable the user to establish a correct mental model of the rela- 
tion between the shared I/O devices and the different functions that they con- 
trol. 

However, a further possibility is to use real-time centralised HMI management 
to resolve conflicts between applications, e.g. actions presented simultaneous- 
ly. A key issue here which types of information that can be presented simul- 
taneously and in which cases conflict resolution is needed, e.g. by means of 
prioritisation or a changing the output modality. These and other issues will be 
investigated in the forthcoming virtual prototyping tests in SP3, but also in the 
evaluation method development in SP2. 

HMI Integration also leads to many technological challenges. In particular, 
how can a many-to-many mapping between applications and I/O devices be 
efficiently implemented, and how can we handle the strong requirements on 
re-configurability and scalability. The AIDE architecture [22] has been devel- 
oped in order to satisfy these requirements. In the further work, it is impor- 
tant that the proposed architecture solution is continuously validated against 
these requirements, and discussed with external stakeholders, in particular 
architecture experts from related projects such as EASIS, PReVENT and GST. 



4.4 Conclusion 

The AIDE project integrates, for the first time in Europe, major efforts on driv- 
er behaviour research, evaluation method development and technological HMI 
development in a single multidisciplinary project. The mid-term results pre- 
sented in this paper constitutes important steps towards a future integrated 
and adaptive automotive HMI, both from a technical and human factors point 
of view. However, much hard work remains in order to reach the final objec- 
tive of the project, i.e. the on-road demonstration of the AIDE concept in three 
validated demonstrator vehicles. 
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Abstract 

PReVENT takes a first comprehensive step towards realising the 
vision of a safety zone around vehicles by means of complementary 
safety functions. The most critical accident scenarios are covered by 
the applications developed. The integration aspects of the project 
demonstrate a number of different applications on one platform and 
making a seamless time to collision horizon from tens of seconds 
down to zero. Due to the multi-causal nature of accidents and a cer- 
tain randomness, safety systems with continuous alert and early 
warning are needed. The foundation for requirements are laid in the 
analysis of driver behaviour and accidents, followed by a definition 
of functions needed from a vehicle to prevent the accident from hap- 
pening or mitigating its consequences. The requirements database 
was created and requirements were classified into six main cate- 
gories. A common PReVENT architecture is presented and high- 
lighted with a detailed description of a sub-project focusing on later- 
al safety applications. 



1 Introduction and Background 

The Integrated Project PReVENT takes a first comprehensive step towards 
realising the vision of a safety zone around vehicles by means of complemen- 
tary safety functions. Their subsequent integration into an overall safety sys- 
tem is expected strongly to reduce the number traffic accidents in the future. 
An additional step towards integration is taken by the INSAFES (INtegrated 
SAFEty Systems) subproject demonstrating a number of applications on one 
platform only. 

Another integration aspect of the project is to create a seamless time to colli- 
sion horizon from tens of seconds down to zero ranging from information from 
map navigation and wireless messages to potential danger warnings to colli- 
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sion mitigation (Fig. 1). This time continuum was deduced from the accident 
analysis and in-depth studies showing that both driver state and driver behav- 
iour (accident causes) and accident situation (typology/scenarios) need to be 
known when optimal safety functionalities and applications are planned. 
Especially knowledge of a multiple causation nature of accidents helps in cre- 
ating applications concentrating on driver state and behaviour monitoring and 
communication systems beyond the on-board sensor capability. 



eSafety Systems 



IVIS = ln-vehide Information Systems, 

ADAS = Advanced Driver Assistance Systems 



Driver Assistance Systems (DAS) 



PReVENT applications area 




t = - 1 sec 



t = — 0.1 sec 



Fig. 1. Seamless time to collision horizon by means of different safety 
applications. 



PReVENT focuses on the most critical accident scenarios such as the loss of 
longitudinal and lateral control leading to head-on and lateral collisions, run- 
off the road accidents and turning and crossing accidents. By analysing the 
driver behaviour and accidents, a set of requirements has been derived for 
each of the PReVENT applications. Eventually, the different requirements lead 
to a common PReVENT architecture. 



Three basic layers are identified: perception, decision (application) and the 
action layer. The perception layer uses the sensors that range from onboard 
sensors such as short and long range radars, cameras and laser scanners to 
GPS signal in conjunction with digital maps and even to vehicle-to-vehicle and 
vehicle-to-infrastructure communication. In terms of enabling technologies, 
the project concentrates mainly on onboard sensors and digital map interface 
for ADAS applications. The perception layer performs low-level and high-level 
data fusion and sends a set of abstracted parameter representing the state of 
the vehicle to the application layer. The map interface supports the perception 
by providing a priori information on the known environment. The application 
layer assesses the situation such as the need for a collision warning, the need 
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for a lane keeping function or an emergency braking. It decides what to do and 
passes the decision to the action layer. The action layer issues warnings via an 
appropriate HMI or engages vehicle control actuators such as steering, brakes 
as demanded by the threat level of the potential accident situation. 

The paper is organised as follows: First it is shown how the accident scenarios 
considered in PReVENT lead to the requirements of the active safety functions 
an how they are gathered to form a requirement data base that allows a sys- 
tematic system specification and, later on, lead to their evaluation and assess- 
ment. The next section describes the overall PReVENT system architecture 
that is derived from the requirements and gives examples for their actual real- 
isation. Subsequently it is elaborated how the singular safety functions will be 
integrated into a single platform in the later phases of PReVENT. Finally, a con- 
clusion section discusses the significance and issues in realising the PReVENT 
vision. 



2 From Accident Scenarios to Requirements 

In addition to previous work in intelligent vehicle area, PReVENT safety 
requirements had its impetus from both EU- and national safety programmes. 
The foundation for requirements were laid in the analysis of driver behaviour 
leading to accidents, followed by actions needed from a vehicle to prevent the 
accident from happening or mitigating its consequences (Fig. 2). 




► 




Fatique, distraction, 
alcohol 



Head-on, run-off the road, 
turninq/intersection 



ic^n 



Inform, warn, assist, 
mitiaat 






Integration, seamless 
time to collision horzion 



Fig. 2. Overall process of laying basis for the detailed definition of 
PReVENT requirements. 
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As indicated in Fig. 2, the most frequent accident scenarios are dealing with 
loss of longitudinal and lateral control leading to head-on and lateral collisions, 
run-off the road accidents and turning and crossing accidents. These scenarios 
are analysed, and a set of requirements has been derived for each of the 
PReVENT function fields and systems. After a grouping and classification 
process, a synthesis of the high level requirements covering the driving sce- 
narios from information/warning tasks and manoeuvring tasks, to stabilisation 
and imminent pre-crash situations is formed and a need for a safety zone is 
expressed. 

To manage numerous PReVENT requirements, a requirements database was 
created. Therein, a division of requirements into six different groups - where 
applicable - was followed (Fig. 3). The database presents, as a top-level ele- 
ment, a detailed accident scenario/unwanted behaviour to which the require- 
ment was targeted. The first level describes the 'Functional requirements', i.e. 
it describes the function that the system needs to perform. The second level 
'Technical requirements' lists the technical features of the system needed to 
perform the intended function. The third level, associated with the perform- 
ance of the requirement, specifies in more detail what the system needs to per- 
form to fulfil the intended function. The HMI requirements describe the actu- 
ator performance and details on how the information of the functionality is 
passed to the driver. User requirements describe the requirements in terms of 
usability and acceptance of the application to be developed. Finally, CoP (Code 
of Practise) details the requirements in terms of the system development, test- 
ing and evaluation and in terms of a requirement of reasonable safety when 
using the safety application. 



Subproject 
accident scenario 



Functional 

requirements 

Technical 

requirements 

Performance 

requirements 

HMI 

requirements 

User 

requirements 

CoP 

requirements 



Verification 

Verification 

Validation 

Evaluation 



Evaluation 

Evaluation 



Result 



Result 



Result 



Result 



Result 



Result 



Fig. 3. Organisation of requirements levels for each PReVENT accident 



scenario. 
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1 Req ID 32.2 


Accident Scenario 1 


Description 


Tailgating scenario 


Explanation 


The traffic and environmental parameters priorities are: 
Gender: Male n Female (both priority 1) 

Age group: over 25 (priority 1) 

Day time: Daylight (priority 1), Dawn (priority 2) 
Weather conditions: No adverse 


Verification / validation / 
evaluation 


Pilots addressing LRM application will be performed upon these accident scenario and the 
combinations emerged from the defined traffic and environmental parameters. 


!■ Rea ID 32.2.1 


Functional req 's 1 


Description 


LRM (Lateral and Rear Area Monitoring): The application is intended to serve as a 
complement and extension to the rear view mirrors, and will enable the driver to detect and 
track vehicles in the rear and lateral area. 


Explanation 


The position, velocity vector and geometrical extents (length, width, height) of vehicles in the 
rear and lateral area will be displayed to the driver. The driver shall be able activate/deactivate 
the active parts of the output, l.e. sound/flashing and other attention attractions shall be 
controllable (on/off). 


Verification / validation / 
evaluation: 


Subsystem and component test in laboratory and simulator. Full function test with CRF test 
car on test track or real roads. 


H Rea ID 32.2.2 


Technical req 's 1 


Description 


The following tasks can be identified for the LRM-function: 

(Tl) Measure ego-speed and yaw-rate. 

(T3) Measure object movement in sensor-range and characterise object dynamics in terms of 
object position & speed (lateral and longitudinal), and distance to the closest point. If possible 
other parameters like boundary points, etc. could also be estimated. 

(T5) Measure driver actions, e.g. direction indicator, steering wheel angle. 

(T9b) Identify objects of interest(e.g. moving / non moving). 

(T12b) Display objects of interest. See also requirements described in 32.1.2.3. 
i Long range radars: a detection range of at least 24m is desired but this is sufficient only for 
low relative speeds; 

Assume a relative speed of 100 km/h, assume a breaking force of 0.6g, the time necessary for 
speed adaptation is approximately 5s, with an extra second to accompany for reaction time it 
becomes necessary to have a window of 6s or more in order to give fast approaching cars a 
fair chance to avoid collision: 6s translates to 170m at a relative speed of 100 km/h. 


Explanation 


1 N/A 


Verification / validation / 
evaluation: 


The false and missing alarms/information will be recorded either in instrumented vehicle(s) 
raw data or in the simulator log files. 


|H Req ID 32.2.3 


Performance req's 1 


Description 


A reliability factor of over 95% and an overall false display of less than 5% in total are basic 
requirements. 


Explanation 


N/A 


Verification / validation / 
evaluation: 


The overall system performance will be evaluated during the LS Pilots. LRM will be validated 
in a simulated environment and the CRF test car. Cross-check of the objects simulated and the 
objects in real environment will be the main validation criterion. 


M Rea ID 32.2.4 


HMI req's 1 


Description 


LRM HMI has the purpose to inform the driver of the surrounding traffic situation in the 
lateral and rear-end area of the ego-vehicle. The LRM HMI should 
l be easy to understand, 
l have intuitive symbols, 
i be non disturbing but available when needed. 

l have different modes (colour, shape) for at least non-critical and cautionary situations. 


Explanation 


N/A 


Verification / validation / 
evaluation: 


HMI will be evaluated in terms of user acceptance in the final Pilots. 


H Rea ID 32.2.5 


User req's 1 


Description 


Success will be assumed if the final limit of LS applications acceptance and usability remains 
above the average (acceptance limit) and the rate of answers is in accordance to the plan 
(i.e. 15-20 questionnaires per Pilot site and application). (Success rates are defined upon the 
user acceptance indicators) 


Explanation 


In terms of the user acceptance assessment, usability, user acceptance and drivers! workload 
while driving with the LRM applications will be evaluated. 


Verification / validation / 
evaluation: 


User acceptance will be assessed via a pre-pilots questionnaire and a post-pilots questionnaire 
on user acceptance, a short questionnaire on usability in both on-the-road and simulator Pilots 
and a drivers! workload questionnaire. 



Tab. 1 . Extract of the PReVENT requirements and evaluation data base for 
the accident scenario 'tail-gating' within the subproject LATERAL 
SAFE 
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The Tab. 1 exemplifies how the requirements database is structured and how 
the contents are described in detail (Tab. 1). A full description and discussion 
can be found in the public deliverable oequirements for preventive safety 
applications [1]. 



3 From Requirements to System Architectures for Preventive Safety 
Functions 

3.1 Integrated Safety High-Level Architecture Principles 

The creation of different architecture and specification levels requires first an 
agreement on an overall picture across the different activities in the EUCAR 
Integrated Safety Programme. In a draft version, a vehicle-centric layered log- 
ical architecture has been split into three environments between which infor- 
mation can flow: the "far environment", specially from the point of organising 
information and services running on the infrastructure, linked to the "near 
environment" from the point of local communication (V2V and V2I) and envi- 
ronmental sensor arrangement leading to a large set of information made 
available to the "vehicle environment" for accurate and reliable perception 
(vehicle dynamics sensors), decision and actuation (HMI). In Fig. 4, the inte- 
grated safety architecture is depicted. 




Fig. 4. Integrated Safety high level architecture: main blocks. 
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This basic decomposition of the architecture is quite general and applies to 
applications beyond PReVENT. It covers the scenarios found in the integrated 
projects GST, AIDE and EASIS. Together with PReVENT, this group of projects 
form the EUCAR integrated safety programme. Also applications that are 
purely based on telematic functions, such as e-Call, traffic jam information, 
floating car data, hot spot warning and speed limitations are encompassed. 



3.2 The PReVENT Three-Layer Approach 

Although the specific requirements necessarily differ for each individual safe- 
ty application, they lead to a common overall PReVENT architecture. 
Reflecting from the EUCAR high-level architecture, three basic layers have 
been identified within PReVENT the perception layer, the decision (applica- 
tion) layer and the action layer (Fig. 5). This basic layered architecture, depict- 
ed below was developed by the PROFUSION subproject and is described in 
more detail in the public deliverable 'Compendium on sensor data fusion' [2]. 




Fig. 5. PReVENT layers reflected from the high-level integrated safety 
architecture. 

The perception layer uses the sensors that range from onboard sensors such 
as radars, cameras and laser scanners to GPS signal in conjunction with digi- 
tal maps and even to vehicle-to-vehicle and vehicle-to-infrastructure commu- 
nication. PReVENT applications rely mainly on the use of environmental sen- 
sors. 

► The perception layer performs low-level and high-level data fusion and 
sends to the application layer a set of abstracted parameters represent- 
ing the state of the driver, the vehicle, and the environment (DVE). 

► The application layer detects and assesses dangerous situations such as 
the need for collision or lane deviation warning or even emergency 
braking. It decides what to do and passes the decision to the action 
layer. 
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► The action layer issues warnings via an appropriate HMI or engages 
vehicle dynamics actuators such as steering, brakes as demanded by 
the threat level of the potential accident situation. 

Typical features of the PReVENT architecture can be characterised as having: 
modularity with add-on options, scalability with retaining the same architec- 
ture in different systems and common information bus such as CAN. 

The various components that are used in the PReVENT applications, are shown 
in Fig. 6. They are highlighted. Each PReVENT subproject with its safety func- 
tions uses a subset of components in each layer needed to achieve its goal. 
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Fig. 6. PReVENT detailed view on perception, decision and action layers. 



3.3 Purpose of the System Architecture: Lateral Control as an Example 

In order to illustrate the purpose of the system architecture, we consider one 
of the PReVENT subproject in more detail: the LATERAL SAFE subproject. 

This subproject introduces a cluster of safety applications of the future vehi- 
cles preventing lateral and rear related accidents and assisting the driver in 
adverse or low visibility conditions and blind spot areas. It covers all other 
directions around the vehicle without regarding the forward area. The appli- 
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cations make up a common multi-sensor platform and include: (i) Lateral colli- 
sion warning system, (ii) Lane change assistant system (Lane Change Aid) and 
(iii) Lateral and rear monitoring system. The problems addressed in LATERAL 
SAFE are illustrated by the three-layered approach of the system including 
sensors, perception, applications and action (with Human-Machine Interface). 

In the following sections, the individual layers are examined in more detail. 
The following architecture components are used within LATERAL SAFE. They 
are extracted from the public deliverable on specifications and architecture for 
PReVENT safety functions [3]. 



3.3.1 Perception Layer 

The role of the perception layer is to provide a physical environment model to 
the application that decides and act s according to the defined functionality. 
The first step is to achieve a common temporal and spatial reference, trans- 
forming raw sensor data into a consistent set of units and coordinates; as a sec- 
ond step the perception layer interprets observations to detect objects and 
extract features in the current physical environment. 

The Long Range Radar, the networks of Short Range Radars (one network on 
each side) and a set of cameras in Fig. 7 deliver signals and images to the per- 
ception layer, from which a description of the environment of the vehicle is 
derived and used by applications to decide appropriate actions towards the 
driver, implemented through adequate HMI. 

The perception part also includes the processing blocks devoted to LRR, SRR 
and Vision signal and image processing, plus an additional Fusion block in 
charge of merging whatever information comes from the above processing 
blocks into a sound description of the environment. Vehicle sensor data are 
also available to all modules. This description of the environment is used in 
turn by three different applications Lane Change Assist (LCA), Lateral Collision 
Warning (LCW) or Lateral and Rear area Monitoring (LRM) which decide which 
controls should be sent to HMI in order to warn/inform/alert the driver of the 
current situation when found pertinent. 

In conclusion the role of the perception layer in LATER A SAFE is to: 

► Carry out "perception enhancement" tasks independent of the applica- 
tion (generic) 

► Describe in a formal way the environment and the traffic scenario 
(semantics) i.e. the current state of the objects, the driver, the vehicle 
and the road. 
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► Support LATERAL SAFE functions under request (specific) 

► Act as a gateway between sensor systems and applications with defined 
interfaces and I/O protocols. 
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Fig. 7. Perception layer of LATERAL SAFE. 



3.3.2 Decision/Application Layer 

Based on the outputs of the perception layer, LATERAL SAFE develops three 
applications: 

► Lane Change Assist (LCA) function: The application has two aspects. It 
informs the driver about presence of vehicles in the blind spot e.g. if the 
direction indicator is set, and it issues collision warnings in case of crit- 
ical lane change manoeuvres of the ego-driver. 

► Lateral Collision Warning (LCW) function: The LCW function will be able 
to warn the driver in cases of objects (e.g. cars, bikes, and infrastruc- 
ture) getting very close to the side of the ego-vehicle 

► Lateral and Rear Area Monitoring (LRM) function The application is 
intended to serve as a complement and extension to the rear view mir- 
rors, and will help the driver to detect and track vehicles in the rear and 
lateral area. 

The complexity of this first integration approach is illustrated in Fig. 8. The 
decision layer recommends the type of action needed in the event of a dan- 
gerous situation or inappropriate behaviour. This is determined by the type of 
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the perceived hazard and the previous interaction with the driver in form of 
information or warnings. Naturally, in individual applications, the decision con- 
cerns only the case whether to interfere by any means in driving or not. 
However, in the integrated approach, the decision means also a judgement 
upon the type of actions needed going from information to intervention. 
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Fig. 8. Decision/ Application layer of LATERAL SAFE. 



3.3.3 Action Layer 

The HMI Manager is responsible for decoding the outputs from the different 
applications. It also makes the decision on which and what information to con- 
vey to the driver and for how long a warning should be active. The HMI CAN 
controller, sound controller and graphics controller are modules that are 
responsible for the interface between the HMI software and the hardware. 

The CAN controller is responsible for the CAN communication to and from the 
vehicle ECU. The Sound controller is responsible for activating the acoustic 
HMI. The Graphic controller is responsible for the graphic HMI. 

Fig. 9 gives a schematic overview of the different modules involved in the 
action layer. The figure contains both software and hardware components and 
how they are connected. 




418 I PReVENT 



Action 



Switches 






HMI manager 



CAN Ctrl 

Sound Ctrl 

Graphics 
Ctrl 



Visual HMI 



- Acoustic HMI 



Display 



Fig. 9. Action layer of LATERAL SAFE. 



3.3.4 HW & SW Platform 

The following Fig. 10 illustrates how the different architectural building 
blocks discussed above are mapped onto the actual hardware architecture of 
LATERAL SAFE. 
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Fig. 10. Hardware architecture of LATERAL SAFE. 
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4 Towards an Integrated Preventive Safety 

The PReVENT integration is further supported by horizontal projects aiming at 
both common use of sensors and vehicle positioning for different applications 
as well as common forum for validating the project work and results. As 
explained earlier, the first step towards a real integration is the new horizon- 
tal project INSAFES that will concentrate on showing the potential of inte- 
grating a number of safety applications on one platform. INSAFES will take a 
step beyond safety applications, targeting specific functions to an integrated 
system, covering a number of applications that are developed within 
PReVENT. In particular, INSAFES addresses the following different scenarios 
and use cases: 

Enhanced longitudinal control functions 

► Integration of Frontal collision warning and Collision Mitigation strate- 
gies - Semi-autonomous braking intervention: braking action confirmed 
by the driver on the brake pedal after a warning. 

► Warning and intervention strategies optimised according to the data 
received from WILLWARN (e.g. about the friction parameters and the 
danger situations further ahead). 

Enhanced lateral control functions 

► Improvement of lane keeping support, optimised also w.r.t. the pres- 
ence of lateral obstacles (e.g. close guard-rails or parallel vehicles) 

► Side collision avoidance/mitigation for trucks within the ego-lane, typi- 
cally in drifting situations. 

Integration and synergies of Lateral and Longitudinal control applications 

► All-around warning function, based on the reconstruction and the 
assessment of all-around traffic scenario. 

► Improved Frontal collision warning/mitigation: Optimisation of the 
warning/intervention strategies, based on computation of a feasible eva- 
sive trajectory, taking the all around scenario into account. 

► Collision Mitigation- Autonomous intervention: optimisation of the 
intervention strategies (for instance switch to semi-autonomous modal- 
ity) depending on the surrounding detected traffic (e.g. evasive 
manoeuvre if feasible, close rear obstacles,...). 

INSAFES goal is to find a methodological approach for functional integration, 
which should result in a higher level of functionality without introducing too 
many interconnections and dependencies between the different functions. 
The functions need to be integrated in order to gain performance and accept- 
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ance but they need to be modular to maintain a simple and flexible solution. 
The key is to find the balanced level of integration. 

Starting from a number of stand-alone functions the three layered architecture 
is basically multiplied (Fig. 11). If no coordination at all is performed, it is like- 
ly that the different functions try to use the same output device (for example 
the brake system) at the same time. Due to the different perception modules 
it is also possible that the different functions don't have the same perception 
of a certain traffic situation. 




Fig. 11. Stand-alone functions with PReVENT functional architecture. 



The easiest way to ensure that the different functions do not try to use the 
same HMI simultaneously and that warnings and actions are transferred to 
the driver in a coordinated way is to introduce some kind of output coordina- 
tion (Fig. 12). This is one of the topics examined by the integrated project 
AIDE, outside of PReVENT. This solution has the advantage that it can be used 
both for functions that are completely stand-alone or functions that use a com- 
mon perception. Within ProFusion2 integration within the perception layer is 
handled. Through this common perception framework the different functions 
get a more unified view of the situation. 
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A solution with a common perception and a coordinated uses a common view 
of the traffic situation and ensures that use of HMI units is coordinated. 
However, no major increase in functionality is obtained (other than what is 
expected from the forming of a unified perception). 

The INSAFES approach to obtain a higher level of functionality is to use both 
a coordinated output and to integrate the functions themselves for a common 
decision. Through the common decision, a new functionality can be added. 




Fig. 13. Functions with a common decision. 



The INSAFES project will thus develop a common decision layer and study the 
interfacing of different functions. One challenge will be to find a level of inte- 
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gration that results in an increased performance but still enables most of the 
different function components to be re-used. The IN SAFES goal is to integrate 
already existing function components, and not to re-develop them or to create 
new ones. 

Also, the action- and output coordination will be covered, in close cooperation 
with the AIDE project. Fig. 13 illustrates the idea of introducing a common 
decision layer and also shows how the action and output coordinator still is 
needed if there are functions that are not fully integrated in the common deci- 
sion concept. 



5 Conclusions and Discussion 

As described, the vision of PReVENT is to create a safety zone around a vehi- 
cle by developing and realising a set of complementary safety functions. The 
justification for this vision comes from the previous work in active safety and 
ADAS area. It has been fragmented and lacking integration that would help 
the task of assisting drivers within one holistic system rather than by a num- 
ber of isolated applications causing problems on HMI-level, at the latest. 

PReVENT is actually a logical continuation and an outcome of pervious work 
carried out in the former Framework Programmes and activities of automotive 
industry, suppliers and academia in the field. Moreover, there has been an 
increasing pressure from governments and EU to address the safety issue. 
Traffic safety work needs more vigour throughout Europe both in terms of 
implementing conventional and introducing new safety technologies. 

Accident analysis, accidentology and review of driver behaviour theories has 
brought forth a number of useful facts when planning technical systems for 
driver assistance. The conclusion from these different analysis and findings is 
that, overall, drivers need a system with continuous alert. It is characteristic 
of any given driver to make occasionally mistakes that in certain conditions are 
fatal. Moreover, an accident to materialise, other contributing factors are often 
needed to trigger the harmful event. Furthermore, when being aware of driv- 
ers' typical time to react changes in his/her environment, it can be concluded 
that safety systems need to 'see' and above all, perceive more and further 
away than what the driver is able to. 

The combination of drivers' reaction times and typical travel speeds in free 
flow conditions is such that that it is useful to have systems able to see even 
beyond a driver's field of vision. The most typical characterisation of an acci- 



Accidentology as a Basis for Requirements and System Architecture of Preventive Safety Applications I 423 



dent situation by survived parties is that it happened "all of a sudden" or very 
quickly The answers reflect the fact that the driver had very little or no time 
to react, make a manoeuvre to escape the situation. For this reason, the basis 
of all safety systems that are in the first instance informative in a way prepar- 
ing drivers for possible incidents along the way ahead. These systems should 
be able to collect information at the minimum, several seconds ahead of the 
driver to allow her/him to process the information and adjust behaviour 
accordingly. Eventually, any given safety system should, however, be more 
than serve as an information system only. 

These systems need to be complemented by assistive functions aiming at cor- 
recting driver behaviour if - despite the pre-information, the driver does not 
change the behaviour. Most of safety applications under development have 
been targeted to adaptive functions. Knowing drivers' tendency to behave not 
by optimising safety but rather guided by other motives, there is still a great 
safety potential in this area. Recently published different accident studies on 
the safety impacts of vehicle stability systems support this assumption. 

Finally, the third cluster of safety applications targeted to by PReVENT are sys- 
tems to mitigate collision consequences. The sub-projects in concern have 
shown convincingly the assets of the collision mitigation systems such as 
reversible and irreversible safety systems to prepare the existing safety sys- 
tems for a collision, active hood and bumper for the protection unprotected 
road users, brake pre-conditioning and finally, fully autonomous braking. 

These applications of basically three different types are being developed in the 
four functional areas all having distinctive features: Safe speed and safe fol- 
lowing, Lateral Support and Driver monitoring. Intersection safety and 
Vulnerable road users and collision mitigation. 

Within PReVENT almost solely safety related applications are addressed. For 
this reason, there is a strong focus on safety validation throughout applications 
and functional areas. Especially, ADAS systems with new models of driver- 
vehicle interaction require a comprehensive validation of safety issues, not 
only the technical safety of systems but also the "safety of usage". 

This question also includes the issue of how long the driver should be kept in 
the control loop. So far, the principle is that the driver will be controlling the 
vehicle as long as possible. Should the situation be allowed to develop so far 
that without vehicle's active safety system interfering in driving, the control 
of the driver will not be taken until at the last moment by the autonomous 
emergency breaking. Or should it be done earlier? Where is the point on time- 
axis, when the driver behaviour should be interfered? Today, there is no clear 
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answer to this. Different alternatives to keep the driver in the loop are being 
studied in Integrated Safety Programme. Above all, this issue is a user require- 
ment. Currently, for example the increasing use of vehicle stability systems 
suggest that to some extent drivers allow control being taken momentarily off 
their hands. Whether this is a case with other active or proactive safety sys- 
tems, is still unresolved. It is possible that if the system is felt assisting driv- 
ing rather than restricting driving, the drivers accept the interference. This 
may be explained by the time the system is interfering in driving. 

Even though, the above is a fundamental issue when decisions are made con- 
cerning the precise functions of various safety applications, eventually, it is a 
good starting point to let drivers themselves to decide on the application and 
function mode they want. 
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Abstract 

The paper introduces the ADAS Horizon concept and the first results 
from MAPS&ADAS project (PReVENT IP) of a standardized interface 
between ADAS applications and digital map data. As a predictive 
sensor called ADAS Horizon, in-vehicle digital maps are an impor- 
tant source of information providing look-ahead capability for ADAS 
applications and providing further information for on-board sensors 
to enhance environment perception. Therefore digital maps have 
large potential for road safety to enhance or enable preventive and 
active safety applications by extending driver horizon. 



1 Introduction 

The development of Advanced Driver Assistance Systems (ADAS) and, more 
generally, of in-vehicle ITS applications which support the driver in driving 
safely, comfortably and economically, are of major importance to the automo- 
tive industry. Typical examples of ADAS applications are Adaptive Cruise 
Control (ACC) or Adaptive Front Lighting System (AFS). ADAS currently per- 
form their function on the basis of information generated by sensors observ- 
ing the vehicle's environment. There is a significant potential for the use of a 
digital map and the vehicle's position to predict the road geometry and to 
track related attributes ahead of the vehicle. ADAS applications can benefit 
from this potential, and new functionality may likely be enabled. In particular, 
ADAS applications will use map data for recognising road infrastructure con- 
ditions at the vehicle's current position, and for a preview along the track 
ahead. 
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Fig. 1. ADAS Horizon concept 



First approaches of map-supported ADAS entering the market will probably 
use the vehicle's navigation system as the map data source for this preview. 
The advantages of using the navigation system are that it already has a stored 
record of the map data, and it already performs the tasks of vehicle position- 
ing and map matching. These functions can readily be used also for the ADAS 
preview. However, each navigation system stores map data in a system-specif- 
ic format, and uses its own version of vehicle positioning and map matching. 
Moreover, when tailored to single ADAS applications, the variety of different 
map interfaces may expand even more - not only due to different data 
sources, but also due to application-specific differences. By standardising 
access to the map data, irrespective of the data provider and physical storage 
format, and by standardising the way that position and track ahead informa- 
tion is presented to the applications, ADAS software can focus on performing 
its main task without having to deal with the complexities of map representa- 
tion. 

The absence of a common standardised interface offering map access to ADAS 
applications has already been perceived in the industry community and has 
led to the foundation of an industry forum where ideas and requirements 
regarding such an interface have been deeply discussed. The extensive dis- 
cussions and work already performed by the members of this group, known as 
ADASIS Forum [1], served as the basis for the MAPS&ADAS subproject part of 
the PReVENT Integrated Project [2]. Requirements and concept proposals dis- 
cussed there among automotive OEMs, suppliers and map providers have 
already reached a certain maturity, stating the importance and close to mar- 



ADAS Horizon - How Digital Maps Can Contribute to Road Safety I 429 



ket position for the MAP&ADAS subproject goals: To develop, test and validate 
a standardised interface able to serve a spectrum of different ADAS-applica- 
tions with map preview data. 



2 Functional Architecture 

Map data access for most ADAS-applications normally use only a small map 
extract around the current vehicle position. This ADAS-specific map extract is 
called the ADAS Horizon (AH) [3]. As shown in the Figure 2, the ADAS Horizon 
provider (AHP) is the entity responsible for the interactions with and trans- 
missions to the ADAS-applications. The positioning module provides a map- 
matched position to both the navigation and the AHP. Using this position, the 
AHP extracts the necessary information from the navigation map and aggre- 
gates it into the ADAS Horizon (AH) which is sent to the application side 
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Fig. 2. Transformation and flow of map data from source via bus towards 
application 



The map preview extracted by the ADAS Horizon Provider (AHP) is accessible 
by all map-supported ADAS applications through the vehicle bus interface. 
Incremental changes transmitted by the AHP are stored on application side by 
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the ADAS Horizon Reconstructor to build the lD-track preview delivered to 
the ADAS application by means of an API. 



3 Functional Structure on Application Side of ADAS- Interface 

The ADAS-interface specification defines the content and organisation of a 
data stream which is sent by the AHP to all ADAS applications [3]. In a simple 
(near future) set-up with only one ADAS-application, this data stream may be 
dedicated; that means it only contains a configuration for data relevant to that 
application. In an advanced far future stage, several ADAS Applications may 
be listeners to the same data stream (e.g., transmitted via CAN bus). In this 
set-up, every ADAS application filters out the data it needs from this stream 
and stores it locally. If the AHP knows the individual needs of all listening 
ADAS applications, it can adapt the stream accordingly. 

The data stream from AHP to ADAS applications consists of: 

► Horizon map elements: parts of the road network with attributes 
together with additional content 

► Vehicle position information together with driving dynamics 

► Metadata 

For the internal architecture of the receiving side, passing the data stream to 
the ADAS-application, see the Fig. 3. 
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application 



Fig. 3. Architecture layout on application side of ADAS-interface 
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The ADAS application will receive the data-stream and handle it in the fol- 
lowing way: 

► In the multi-application environment the horizon map elements are first 
filtered: only the road parts and attributes that are needed by the ADAS 
application are passed. For a single application scenario, this filtering is 
probably not necessary as according the configuration as only required 
data is transmitted. Then the horizon data store manager will store the 
filtered horizon map elements in the horizon map data store, assuming 
there is free storage space available. If there is no space, it will try to 
remove old elements that are no longer needed because the vehicle has 
advanced beyond them. For this, a mechanism to manage the available 
storage space is to be implemented. 

► From this data store the relevant part for the application can be recon- 
structed - the left hand block of the application is called Horizon 
Reconstructor (AHR) for this reason. The AHR passes the relevant data 
to the application, generally in the convenient lD-format for track pre- 
view functions, confined to its application-specific contents. The selec- 
tion of the application contents should be performed to a maximum on 
the bus controller side (using e.g. message identifiers) and to a mini- 
mum by software mechanisms discarding non-relevant message parts 
in the application itself. 

► Besides the basic geometry there is information that is continuous with 
respect to the along track distance (e.g. curvature) which can be mod- 
elled as profile along the track. 

► The vehicle position, speed, heading, etc. is stored inside the AHR mod- 
ule of the application as the last known position, together with a time 
stamp. If the accuracy of the 'last known position' is sufficient for the 
ADAS application, it can use it directly for look-ahead calculations. If the 
application needs more accurate data, information about the delay of 
the transmission of the position information is provided by using time 
stamp or age information. Based on the last transmitted position and its 
time stamp, the position extrapolator may compute an extrapolated 
vehicle position with higher accuracy at any time, independent from 
the rate it is transmitted from the AHR For the extrapolation it is also 
required that there is some basic prediction of the most probable path 
in order to provide preview across junctions. 

Metadata is handled by the controller. Depending on the type of Metadata, it 
is passed actively to the respective module to influence functionality, or it is 
stored in a Metadata store for access by the modules. 
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3.1 ADAS Interface Implementation 

Several components and systems have been developed and implemented with- 
in MAPS&ADAS to enable test and validation activities [4]. 

► An ADAS Horizon Provider (AHP) is a system that generates an ADAS 
Horizon and distributes it on a bus. Four AHP systems have been devel- 
oped, 2 dedicated by Navteq and Navigon and 2 navigation based sys- 
tems by Siemens VDO and Blaupunkt. 

► An ADAS Client is receiving the ADAS Horizon from the bus. It includes 
an ADAS Horizon Reconstructor (AHR) that reconstructs the received 
ADAS Horizon, and makes its content available to the application via an 
API. 

► The ADAS interface has been implemented in a sample ADAS applica- 
tion: Active Cruise Control. The implementation in the BMW ACC is 
based on an existing ACC, using the reference implementation of the 
AHR. 

► A reference implementation of the AHR was developed for testing 
ADAS Horizon Providers systems and as the basis for the implementa- 
tion of the ADAS horizon in an ADAS client. 

► The Test Container is a separate system that can measure the impact of 
the protocol on the CAN bus for assessment. 
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3.2 ADAS Interface Testing 

The test container provided by DC was intended for testing and evaluating the 
mutual influence of the original data traffic present on a vehicle CAN bus and 
the ADAS Horizon data stream injected onto this bus [5]. Mutual influence 
means how the characteristics of latency and transmission error statistics of 
one data stream are influenced by the presence of the other. 

For these tests, the typical background CAN traffic of a vehicle that might use 
map supported ADAS function must be recorded and later replayed on a CAN 
bus injecting a corresponding data stream for transmission of horizon data 
using the ADAS interface. The reason why this must be done off-line in a 
'sand-box' environment is, that it is not possible to introduce extra data on the 
CAN of a series vehicle under normal operating conditions (i.e. driving on pub- 
lic roads) without incurring danger of vehicle malfunction and safety risks. 

Consequently, the tests involving real vehicle CAN data streams were per- 
formed by the four OEMs (BMW, DC, Ford, Volvo). On the other side, the four 
supplier companies (Blaupunkt, Navigon, Navteq, Siemens-VDO) helped by 
producing the ADAS interface data streams within the different vehicle CAN 
environments. 




Fig. 5. Test architecture layout for online monitoring of MAPS&ADAS 
data stream and background data traffic on CAN bus 

Of course it is interesting for each OEM how an implementation of the ADAS 
interface performs on his vehicles and how it does influence the existing CAN 
traffic. For this reason, the main test task for all the OEMs was to evaluate the 
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OEM specific properties of a possible interface usage on the own cars. Also, the 
OEMs were entitled to provide the background CAN traffic files for general 
testing, as they are able to produce relevant real life CAN traffic examples. 

There are a number of factors that influence the behaviour of the 
MAPS&ADAS transmission within an existing background CAN traffic. These 
factors on one hand originate from the background traffic, into which the 
MAPS&ADAS transmission is to be injected. On the other hand, they are influ- 
enced by the properties of the injected transmission itself. 



3.3 ADAS Interface Validation 

As one of the major the objectives of MAPS&ADAS, the definition and the 
development of an interface between in-vehicle map data sources and assis- 
tance systems is of pure technical nature, the technical validation of the inter- 
face properties is the most important issue of the validation task [6]. 

However, there are properties, which obviously do not need to be examined, 
as they are known to be uncritical from state-of-the-art. On the other hand 
there are interface properties whose outcome is unsure at the moment of its 
definition, but which may be very important for success or failure of a market 
introduction of such an interface. 

Consequently, the technical validation performed within MAPS&ADAS is 
focussed on these most critical or most important issues: 

► Bandwidth usage: Does the interface as proposed by MAPS&ADAS effi- 
ciently use transmission bandwidth for standard application require- 
ments? E. g. what bandwidth is spent for a typical ACC preview, con- 
sisting of geometry and road class information? 

► Error properties and consistency of transmitted data: Is the interface 
transmission and interpretation error level sufficiently low? What level 
of Integrity (error recognition rate) can be reached? 

► Timing Issues: How long does it take for the horizon and positioning 
data to be transmitted? How 'old' is the image presented to the appli- 
cation? 

► Position and distance-to-object accuracy: How does the timing influence 
the knowledge and accuracy of the proper vehicle position and its rela- 
tion to the event points of the preview? 

► Interoperability: Is it assured that different horizon provider software 
solutions, based on different navigation systems and map databases 
deliver a horizon data stream fully compatible to the interface specifi- 
cation? 
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4 First Test Results 

Test drives have been performed on different vehicles in different environ- 
ments and the resulting data streams been mixed offline while varying sever- 
al transmission parameters [5]. Of special interest were error and latency 
behaviour of the MAPS&ADAS data stream itself and also its influence on the 
background traffic which is already present on that bus for normal vehicle 
control purposes. The whole variety of performed tests, even the most chal- 
lenging set ups have shown the following first results: 

► Additional bandwidth requirement is normally around 0.1% of the 
busses capacity reaching up to 1.5% for very demanding set ups. 

► The maximum added latency charged to the background traffic (lower 
priority CAN frames) keeps below 0.37 ms for normal conditions and 
below 0.53 for more challenging setups. 

► The total latencies experienced for the MAPS&ADAS data stream are 
around 0.25 ms (average) and 0.5 ms maximum for normal conditions, 
reaching up to 0.5 ms average and 1ms maximum for more demanding 
test conditions. 

► The transmission properties of the more flexible, more general 
MAPS&ADAS protocol compared to the existing, dedicated BMW pro- 
tocol are comparable. This is on one hand valid for the bandwidth usage 
figures and on the other hand regarding latency times and influence on 
background traffic. 




Fig. 6. Examples of ADAS Horizon visualization from Daimler Chrysler, 
Navteq and Navigon 



5 Conclusions 

The encouraging first results achieved within MAPS&ADAS assess the techni- 
cal feasibility of a standardised interface between ADAS applications and dig- 
ital map data and therefore confirm the potential of digital map for road safe- 
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ty to enhance or enable preventive and active safety applications by extend- 
ing driver horizon. As a predictive sensor called ADAS Horizon, in-vehicle dig- 
ital maps are an important source of information providing look-ahead capa- 
bility for ADAS applications and providing further information for on-board 
sensors to enhance environment perception. 

These expected benefits of the ADAS Horizon concept are reflected by the 
increasing interest and support from the automotive industry. Since 2002 
major industry stakeholders have joined the ADASIS Forum (26 Members from 
vehicle manufacturers, ADAS suppliers, navigation system suppliers and map 
providers) to promote and support implementation and near future market 
introduction by means of standardisation. The main direct impact of a stan- 
dardised interface will be on the economic side. This impact may be roughly 
estimated in terms of economy on development efforts. 
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Abstract 

In the 6 th Framework Program of the European Commission, the 
Integrated Project PReVENT includes Intersection Safety. The 
INTERSAFE project was created to generate a European Approach to 
increase the safety at intersections. The project started on the 1 st of 
February 2004 and will end in January 2007. 

The main objective of the INTERSAFE project is to 
►improve safety and to reduce (in the long term avoid) fatal 
collisions at Intersections 

In order to identify the most relevant scenarios for accident preven- 
tion, a detailed accident analysis was carried out. Based on the sce- 
narios and the driver mistakes derived from the accident analysis a 
basic functionality is described. It considers for example the time 
budget, which is available in order to warn the driver. 

The importance of these accidents leads to a deeper analysis of the 
scenarios. An in depth analysis of available data from reconstructed 
accidents in France and Germany show the central position of espe- 
cially two accident types: 

►Collisions with oncoming traffic while turning left and 
►Collisions with crossing traffic while turning into or straight 
crossing an intersection. 

Additionally the importance of the actual right of way regulation 
leads to the consideration of traffic light controlled intersections. 
The specification of the key technology components like sensors and 
communication technologies will be derived from the given require- 
ments. Altogether, about 60% - 72% of all accidents in intersections 
are covered directly by the selection of these three scenarios. The 
possible coverage of other comparable accidents needs further inves- 
tigations. This paper will describe the INTERSAFE approach of inter- 
section safety and discuss the functionality to avoid fatal accidents 
at intersections for the chosen scenarios in detail. 
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1 Introduction 

A general overview on the INTERSAFE subproject was given in [1]. The main 
focus of this paper is to give a better understanding of the functionality of the 
sensor based test vehicle used in this project. 

The INTERSAFE test vehicle is equipped with two laserscanners, one video 
camera and additional communication systems. The video camera is used to 
process data about lane markings at the intersection while the laserscanner 
collects data of natural landmarks as well as data about other objects and road 
users. The fused data of the laserscanner and the video camera and a detailed 
map of the intersection (high-level map) are used for localisation of the vehicle 
within the intersection. Together with the object data collected by the laser- 
scanner a broad overall view of the scene is constructed. 

The additional communication module allows wireless infrastructure to vehi- 
cle communication or, in extension, for communication via infra-structure with 
other road users that are equipped with such communication facilities. The 
test vehicle and the mounting positions of the sensors are shown in Fig. 1. 




Fig. 1 . Sensor integration in the INTERSAFE test vehicle 

Due to the fact that this paper deals with the functionality of the application 
in the test vehicle only the laserscanner is considered, because this sensor 
detects and tracks the other objects which are essential for the application 
algorithms. Nevertheless, the video processing data and the fusion with the 
laserscanner data is crucial for the correct functioning of the host localization 
process [2]. 
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In the first section of this paper the laserscanner based object detection, track- 
ing and classification as well as the localization of the host vehicle is described. 
This serves as the input level for the scenario interpretation and risk assess- 
ment which is described in the second part. 



2 Object Detection, Tracking and Classification 

In order to meet the functional requirements of monitoring crossing traffic the 
laserscanner is integrated into both left and right front corners of the demon- 
strator passenger vehicle as shown in Fig. 2. With a scan area of 150° each and 
an overlap between both scanners of 80° a combined scan area of 220° was 
realised. 




Fig. 2. Schematic of the ALASCA laserscanner scan area with the demon- 
strator passenger vehicle (left), laserscanner for detection, tracking 
and classification of road users (right). 



The automotive laserscanner combines a 4-channel laser range finder with a 
scanning mechanism, and a robust design suitable for integration into the 
vehicle. The object data, as illustrated in Fig. 3, is suitable for various ADAS 
applications. 



In the beginning the generated range profile is clustered into segments. 
Comparing the segment parameters of a scan with predicted parameters of 
known objects from the previous scan(s), established objects are recognised. 
Unrecognised segments are instantiated as new objects, initialised with 
default dynamic parameters. 
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Fig. 3. Object recognition using laserscanners: Road users are tracked and 
classified crossing an intersection. The shape of the intersection, 
registered in the feature-level map, is painted grey. The coloured 
dots are measurements of the laserscanners at different objects. 

The host vehicle is drawn as a blue rectangle at (0;0) position. 

In order to estimate the object state parameters a Kalman filter is well known 
in the literature and used in various applications, as an optimal linear estima- 
tor [3]. The Kalman filter was chosen, as it allows for more complex dynamic 
models, which are necessary for precise object tracking. We evaluated our data 
on different association methods, such as the nearest neighbour and the glob- 
al nearest neighbour method [4]. 

Object classification is based on object-outlines (static data) of typical road 
users, such as cars, trucks/buses, poles/trees, motorcycles/bicycles and pedes- 
trians. Additionally the history of the object classification and the dynamics of 
the tracked object are used in order to support the classification performance 
[5,6]. 

In a simple algorithm road users are classified by their typical angular-outline 
using only the geometric data [7]. Additionally the object's history and its 
dynamic data are necessary to enable a robust classification [8]. 
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In case there is not enough information for an object classification a hypothe- 
sis is generated, based on the object's current appearance. The temporary 
assignment is valid, as long as there is no violation of limiting parameters. 
However, the classification is checked every scan to verify the assignment of 
the specified class [9]. 

An environmental model supports the selection of a suitable class. The under- 
standing of the traffic situation may further improve the classification 
resultsln recent projects the scan data fusion was very processing-intensive. 
The laserscanner gathers the range profile in a certain time while the mirror 
is rotating with the host vehicle usually moving. This leads to a shift in the dis- 
tance, with respect to the vehicle coordinate system, of actually equidistant 
targets that are measured in different angles due to the time in scan. Every 
measurement is shifted to a common time base with respect to the host vehi- 
cle's movement. 

Two laserscanners usually measure the same object at different times. If a 
fused scan just combined the two scans the same object would be at different 
places in the vehicle coordinate system. To minimise this effect, the scan data 
fusion is based on synchronised laserscanners to provide a consistent fused 
scan. 

The ECU periodically sends signals (with the desired rotation frequency) to 
both laserscanners. The laserscanners adapt their rotation frequency to the 
synchronisation frequency and the angle of the rotating mirror to the sync-sig- 
nal. The accuracy of synchronisation is about + 2 ms. 



3 Localisation of the Host Vehicle 

GPS-based localisation is unable to provide sufficiently accurate or reliable 
localisation in areas where intersections are located. This is particularly true 
in urban areas where free view to sky is limited. The laserscanner system 
obtains a relative position within the intersection by detecting landmarks such 
as posts and other similar fixed objects at the intersection. 



3.1 Map Building 

In a first stage a grid map of an intersection is generated by moving the laser- 
scanner across the intersection under consideration of the host vehicle's move- 
ment. With known position and orientation of the vehicle at each time step the 



442 I PReVENT 



distance measurement points of the laserscanner are transformed to a local 
map coordinate system. Therefore, the mapped region is discretised into small 
boxes (5 cm x 5 cm) and the number of measurement points in each box (hits) 
is accumulated. 

The visualisation of the resulting grid map (GMAP) is done by standard bitmap 
images. Each pixel corresponds to a specific box of the discretised region. The 
more hits in a box the darker is the pixel in the bitmap image. Fig. 4 shows a 
GMAP of an intersection. 




driven direction during 



rx 



Fig. 4. laserscanner grid map and video reference image. The arrow shows 
the direction the vehicle drove along during mapping process. 

The obtained GMAP is used to determine precisely the coordinates of static 
objects, like posts of traffic lights or traffic signs. These static objects (features) 
are later used as natural landmarks for an host-localisation algorithm. 



3.2 Localisation 

GPS provides approximate position measurements. Using this approximate 
position, the detected natural landmarks are transformed to the laserscanner 
coordinate system. The main challenge is to associate the transformed fea- 
tures to the data of the laserscanner. This is done by the algorithm proposed 
in [10]. After association the position and orientation of the vehicle's position 
is determined with respect to the known positions of the features, relative to 
the intersection coordinate system. 
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4 Functionality in the Test Vehicle 

The following part describes the basic functionality of the application in the 
sensor test vehicle in order to assist the driver for the most relevant intersec- 
tion accident scenarios. These scenarios were identified in a detailed accident 
analysis [11]: 

1 . Left Turn across Path 

2. Turn Into/Straight Crossing Path 

3. Red-Light-Crossing 

Here the functionality for assisting the driver in the first two scenarios is 
described. This includes the scenario interpretation, risk assessment and the 
HMI level. The whole functionality considered within the INTERSAFE project 
including driving simulator and communication modules is described in [12]. 



5 Scenario Interpretation 

As mentioned above, the sensors of the test vehicle permanently observe the 
environment and thus a consolidated view on the current, highly-dynamic 
scene is build. This includes the self-localisation of the host vehicle as well as 
the detection, tracking and classification of the other road users relative to the 
intersection. This allows the determination of all possible conflicts, hence, like- 
ly paths and finally the risk. 



5.1 Conflicts 

In a first step possible conflicts for the host vehicle at the intersection are iden- 
tified. This is based on data from the high-level map. Since in the high-level 
map geometry of the different lanes as well as right-of-way regulations appear, 
it is possible to identify priorities for the traffic flow for each considered inter- 
section. The computation of all theoretical conflicts within the intersection is 
a static process, which solely uses static data of the high-level map and needs 
to be computed only once each time when approaching an intersection. For 
this process it is urgently needed that the high-level map is up-to-date. Wrong 
information about the right-of-way regulations could lead in a total misinter- 
pretation of the scene and therefore in a malfunction of the assistance system 
on the whole. An example of possible conflicts is shown in Fig. 5(a). 
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As soon as the localisation of the host vehicle and the other traffic participants 
is performed, the remaining conflicts are extracted as it can be seen in Fig. 
5(b). These are the conflicts that could appear regarding mistakes of the host 
vehicle as well as the errors of the other vehicles. 

Taking the right of way regulation into account the conflicts for ones own mis- 
takes can be extracted in Fig. 5(c). 

The extracted conflicts are only possible ones since they do not regard the 
time domain. This becomes important in the risk assessment evaluation later 
on. 






Fig. 5. Computation of conflicts 



5.2 Plan Recognition 

The conflicts, as computed above, are not necessarily dangerous if the plans of 
the vehicles will follow the conflict-free paths. Therefore, it is important to per- 
form a plan-recognition in order to identify the possible manoeuvres for each 
traffic participant (including the host vehicle) and their degree of probability. 
The inputs for this process are precise lane information of the high-level map, 
the position and dynamics information of all vehicles and possibly the indica- 
tor information of the host vehicle and the other vehicles (transferred by car- 
to-inf r astructure-to-car communication) . 

Bringing all this information together with the predefined typical occurrence 
and combination of these parameters for a special manoeuvre, a probabilistic 
model for the behaviour of the road users can be extracted (see Fig. 6). 

These probabilistic plans are combined with the conflicts computed in the first 
step. The combination forms a consolidated scenario interpretation for the cur- 
rent scene that serves as basis for the risk assessment. 



Intersection Safety - the EC Project INTERSAFE I 445 





Fig. 6. Simplified diagram of plan recognition 



6 Risk Assessment 

The scenario interpretation provides all possible conflicts and the assigned 
probabilities of the driving behaviour of the vehicles, but, as mentioned before, 
omitting the time domain. The risk assessment takes these probabilities as 
weighting factors and computes a risk level for each possible conflict in time 
and space domain. Here, the time becomes important, because the risk of a 
conflict is highly dependent on the fact whether or not two vehicles are reach- 
ing the conflict point (i.e. a defined area around this conflict point) in the same 
time interval. 




Fig. 7. Schematic diagram of risk assessment for one conflict 
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A fuzzy rule-base is built to reproduce "human thinking" for risk assessment 
so that an adequate strategy can be formulated in an easy way Important 
input variables are speed, acceleration and distance to the conflict. But also 
inputs such as the type of the opponent could be of interest in order to trans- 
late it to a severity index for a possible crash. The results of the risk assess- 
ment module are different risk levels for all possible conflicts in the scene. 

For a first approach the conflict with the highest assigned "risk level" will be 
considered to be the most important one. This conflict is passed to the warn- 
ing strategy level. 



7 Warning/HMI 

A big challenge for humans while driving a car is to assess parameters like 
speed and distance of other approaching vehicles. These parameters are 
directly linked to the evaluation of the risk of a certain traffic situation. In 
order to assist the driver in such a situation an advanced driver assistance sys- 
tem has to estimate the risk level and present it to the driver in an adequate 
way. 

Nowadays driver assistance systems mostly use warning strategies like 
appearing signals or sounds to alert the driver when a situation gets danger- 
ous, i.e. when the computed risk rises over a defined level. The system 
approach used here is a warning interface that visualises the risk level in a 
continuous manner for the time of an identified situation that could become 
dangerous. The driver will have a direct visual link to those parameters that 
are difficult to estimate. For example, if the driver is approaching an intersec- 
tion and intends to turn left, he has to watch out for oncoming traffic. The 
speed and distance of the oncoming cars is one measurement to determine if 
a turn-left manoeuvre is safe or not. Fig. 8 shows an example of our visualisa- 
tion interface for an intended left-turn. 



5 

w 

7T 




Fig. 8. Continuous HMI for visualisation of the risk level 
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During the whole approach the driver has the direct link to the risk level com- 
puted by the system (i.e. momentary value and derivative). The advantage is 
that there is no possibility to get surprised by a flashing light that tells him 
suddenly that a situation gets dangerous. Due to the continuous HMI interface 
he will be prepared for the situation and can estimate the risk for his intend- 
ed manoeuvre more easily by taking his own driving skills into account. Beside 
the risk level visualisation there has to be additional information for which 
conflict the risk level is shown. The idea is to use simple pictures of well known 
traffic signs so that the driver ideally needs no long time to understand for 
what the warning is for. User acceptance tests at the end of the project will 
show the suitability of such an HMI visualisation. 



8 Conclusions 

A comprehensive approach was presented which enables new functions for 
increasing intersection safety by use of an autonomous on-board sensor sys- 
tem. Combining such a system with an additional communication module even 
more applications are possible. Two very precise laserscanner with a broad 
combined scan area of 220° around the front of the vehicle are used to local- 
ize the host vehicle within the intersection using natural landmarks as well as 
to detect, track and classify other road users. With this information and a 
detailed map of the intersection an overall view of the scene can be con- 
structed. A scenario interpretation and risk assessment strategy that tries to 
emulate human way of thinking was introduced. 

For the driver's recommendation it is fundamental not to have just binary 
information, but better a risk level assessment. With our HMI approach, the 
driver can easily assess, if a situation is risky for his driving skills, or not. 
Future work has to ensure that drivers don't use such a risk level based rec- 
ommendation for risk compensation. 
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Abstract 

This publication focuses on a modular architecture for sensor data 
fusion regarding to research work of common interest related to sen- 
sors and sensor data fusion. This architecture will be based on an 
extended environment model and representation, consisting of a set 
of common data structures for sensor, object and situation refine- 
ment data and algorithms as well as the corresponding models. The 
aim of such research is to contribute to a measurable enhancement 
of the output performance provided by multi-sensor systems in 
terms of actual availability, reliability, accuracy and precision of the 
perception results. In this connection, investigations towards fusion 
concepts and paradigms, such as 'redundant' and 'complementary', 
as well as 'early' and track-based sensor data fusion approaches, are 
conducted, in order to significantly enhance the overall performance 
of the perception system. 



1 Introduction 

In the European member states there are about 1.200.000 traffic accidents a 
year with over 40.000 fatalities. This fact points up the growing demand for 
automotive safety systems, which aim for a significant contribution to the 
overall road safety. For this reason a current technology field of the automo- 
tive industry focuses on the development of active safety applications and 
advanced driver assistant systems. These aspire a reduction or at least an alle- 
viation of traffic accidents by means of collision mitigation procedures, lane 
departure warning, lateral control, safe speed and safe following measures. 

The common nominator and key feature of all this novel safety systems is the 
accurate, robust and reliable perception of the vehicle's environment. 
However, current off-the-shelf single sensor approaches cannot always fulfil 
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these challenging demands. Therefore most of these applications base on per- 
ception systems, which process the data from multi sensorial platforms via 
data fusion methods. 

The Preventive and Active Safety Applications project (PReVENT), which is 
part of the Sixth Framework Programme, contributes to the safety goals of the 
European Commission (EC). PReVENT addresses the function fields of Safe 
Speed and Safe Following, Lateral Support, Intersection Safety and Protection 
of Road Users and Collision Mitigation in order to cover the field of active safe- 
ty. The majority of these functions are characterized by using perception 
strategies based on multi-sensor platforms and multi-sensor data fusion. 

Hence, the strategy of PReVENT was to initiate a cross-functional subproject 
called ProFusion in order to streamline and to develop the subject of multi-sen- 
sor data fusion in a greater degree of depth and in a more systematic approach 
as compared to the primarily function-driven subprojects. The role of 
ProFusion, inside PReVENT, is to streamline the sensor data fusion by, e.g. 
gathering requirements, defining certain standards and developing fusion 
algorithms. 

In this context we propose a multi sensor fusion architecture, which facilitates 
the robust and reliable processing of multi sensor data by providing different 
data fusion approaches, a common data structure and a common modelling 
within one framework. Due to the special design of the framework and the 
underlying algorithms it is not limited to a single use-case but can serve as a 
modular platform for environment perception and thus build the basis for dif- 
ferent kinds of safety applications. 

An additional rationale is motivated by the observation, that a variety of 
national and international research projects are devoted to the development 
and improvement of active and preventive safety systems, and that all of them 
are affected by the limited performance and even by deficiencies of the cur- 
rently available sensor platforms. As PReVENT is considered as the core of the 
eSafety research and development initiative, it has been obvious to embrace a 
cross-functional subproject that adopts a variety of challenges and open issues 
in the field of multi-sensor perception. 



2 The Fusion Architecture 

As already outlined in the introduction the perception of the vehicle's envi- 
ronment is the crucial factor of a driver assistance and active safety system. 
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Therefore the major objective of the conducted research is to push the sensor 
data fusion techniques used for automotive environment perception beyond 
the current state-of-the-art ([6], [7]). This will be done by setting up a modular 
and interoperable fusion architecture for multi sensor systems, which inte- 
grates diverse approaches (e.g. low and high sensor data fusion, algorithms for 
situation refinement, etc.). In doing so the proposed framework is not limited 
to any special safety function but operational with all different kinds of appli- 
cations like collision mitigation, lateral control, lane departure warning and 
others. 





DATA STRUCTURE 



S€NSOS . 
DATA STRUCTS 



Fig. 1 . The sensor data fusion architecture 



Fig. 1 outlines roughly the concept and the components of the sensor data 
fusion framework: 



The system will be based on an extended 4D scene / environment model [1]. 
Objectives are to deliver an extended view of the environment to applications 
(in terms of more detailed representations) and to define data structures for 
each processing step during the fusion process (e.g. for raw sensor data, signal 
features, objects detected in the environment). 

For object refinement [2], one will develop existing and new algorithms for fil- 
tering, data association and object classification. Especially, chances for deduc- 
ing confidence information from complementary and redundant sensor data 
for each processing result will be analysed. Additionally, the research work 
will be focused on new motion models. 
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Research topics on situation refinement will include trajectory classification 
and prediction for objects on the road, trajectory estimation of the ego-vehicle 
based on multi source information (e.g. detected lanes, ego-vehicle dynamics 
and map data), as well as on generic decision components for the prediction 
of the driver's intention. 

The newly developed perception system [3,4], and especially the different 
fusion approaches, will be implemented, integrated and evaluated in open-loop 
real-time environments, and ultimately utilised in the closed-loop on-board sys- 
tems of prototype vehicles (owned by BMW, CRF, DaimlerChrysler and Volvo 
Tec). These vehicles are equipped with distinct sensor platforms (including 
stereo vision cameras, FIR cameras, short and long range RADAR devices and 
LIDAR sensors) and serve diverse active safety application [5]. 

The succeeding paragraphs describe the main components of the fusion 
framework, especially the different fusion approaches, in detail. 



2.1 The Early Fusion Approach 

The term "early fusion" is derived from a fusion concept, which is based on 
only slightly pre-processed data. They are processed together from an early 
stage of the processing chain. Such an approach permits the fusion module 
itself to process all data from the different sensors "as a whole". Taking use of 
the redundant sensor information and matching these data to one common 
and consistent scene model of the environment it should be feasible to slash 
inconsistencies and to increase the robustness and reliability of the processing 
results. 

Thereby an early fusion system is working as follows: Several sensors deliver 
raw sensor data, for example echoes from RADAR devices, images from vision 
devices or FIR cameras, etc. to the sensor pre-processing unit. During sensor 
pre-processing specifics of sensor signals are extracted (peaks, plateaus, 
regions with same properties, etc.) and grouped. The resulting features can be 
provided in different abstraction levels. Based on these features objects are 
detected, classified and tracked over time in further processing steps. 

All processing steps make, more or less, use of presumptions related to the 
vehicle environment. These presumptions, stored in models, should be consis- 
tent in the whole processing sequence. Thus beside the one common environ- 
ment model of the early fusion module there should not be used any other 
model in other processing steps. Therefore the input data to early fusion are 
required to be based on the same models or, if not possible because pre-pro- 
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cessing is contained and hidden in sensor components, to be based on minimal 
assumptions. As already indicated before, early fusion does not implicate to 
handle only some first processing steps, but takes data from early processing 
stages from all sensors, fuses and processes them together to one common 
result, the environment description of the vehicle. 



Thereby the early fusion module is internally composed of several functional 
units, which are necessary to put the fusion process into practice. Fig. 2 illus- 
trates the different components of the fusion module and shows their interac- 
tion. It mainly shows the circle of processing steps that have to be executed: 
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Fig. 2. Early fusion functional architecture 

The following paragraphs develop the functionality of the fusion module's 
components in more detail: 

The process of sensor data association or matching is a key challenge of the 
early fusion architecture. Like in other fusion systems feature vectors are 
extracted from the sensor output data. In the matching step these feature vec- 
tors have to be assigned to the respective corresponding objects in the mod- 
elled track history causing the measurement data of the feature vectors. In 
track fusion systems the data association is done based on the pure object data 
from the sensor processing units and the object data from track history. In the 
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early fusion module the data association poses the challenge, as this matching 
step has to be done for data on different level of abstraction, i.e. for edge to 
edge association up to edge to object / track assignment. Similarity measures 
are used to find the most probable and reasonable assignment of the acquired 
sensor data to the data of the object's history Therefore they assess the asso- 
ciation of the features based not only on their (Euclidian) distance, but in con- 
sideration of their covariance information and the origin of the sensor data. 

As the early fusion approach is model driven, a further task of the matching 
module is the attribution and classification, respectively, of sensor data or fea- 
tures to different object models, which provide the basis for the characterisa- 
tion of the environment. In the initialisation phase the matching module gen- 
erates object assumptions for new features based on a set of given (shape and 
measurement) models (e.g. for the road, other cars and trucks, pedestrians, 
etc.) and allots the sensor data to the respective model that proves to be the 
most probable relative to the other ones. With the help of additional sensor 
data the respective object assumptions are either confirmed or discarded and 
newly reassigned over time. Whenever new features from the perception 
devices are available, the system checks if they can be associated to already 
existing objects in the environment description or if they are enough signifi- 
cant to create a new object assumption. 

The handling and management of features, objects and tracks takes place in 
the environmental data structure. In this structure all information on the own 
car, the multi sensor system and the surrounding environment is stored just 
like the relations between these objects. Main task of the object management 
module is the addition, updating, fusing, splitting and deletion of features, 
objects and tracks to/from the environment data structure on the basis of new 
measurement data, data association operations and object hypotheses. 

For the step of state estimation and prediction a realisation of the Extended 
Kalman filter is to be intended due to his simplicity, optimality and tractabili- 
ty in tracking and estimation of (non-linear) systems 

The goal of the fusion module is to provide an as detailed as possible descrip- 
tion of the own vehicle's periphery. This description, which contains all detect- 
ed objects / tracks together with their attributes as well as their spatial inter- 
relation in the surrounding area of the vehicle, can be easily generated at any 
time from the environment data structure. The format of the fusion module's 
output is generic and can be adapted to fit the respective application. 
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2.2 The Multi Level Fusion Approach 

The holistic representation of the environment on the basis of the extended 4D 
environment model and the common data structures allows sensor data pro- 
cessing with the help of a multi-level fusion algorithm. Multi-level fusion 
means that not only one level of fusion exists but data components belonging 
to one physical object are scattered over different levels and evidently fused 
on several different levels too e.g. signal level, feature level and track level. 

In general the chosen level of fusion is object and model dependent. Therefore 
for every object a certain hierarchical fusion strategy can be defined. This way 
the object tracking has inputs from tracked features, untracked features as 
well as from signal level. 

A typical property of multi-level fusion is the use of back loops between the 
levels. Therefore feature models are used to describe which data from which 
level should be used to maintain an object track. This is organized by the multi- 
level fusion management. A special case of multi-level fusion is a processing 
on adaptive chosen levels. This allows the fusion strategy and selection of a 
certain fusion level to be dependent on the actual sensor data and the obser- 
vation situation of an object. That's why a better processing strategy can be 
achieved in most cases. 

The multi-level fusion approach means that several levels of fusion exist. 
Information and data which are generated by a specific object in the real world 
scene are fused on several different levels of abstraction. There are data com- 
ponents on the signal level, the feature level and the track level. At signal level 
all raw or pre-processed data coming from single sensors can be found. The 
data produced by different sensors are forwarded to the processing chain 
where they are processed and fused with data from the same, higher or lower 
level (Fig. 3). 

Multi-level fusion can be performed in two oppositional ways - a bottom-up or 
a top-down strategy. Both strategies can be used at the same time parallel and 
sequential in the processing chain. By performing the bottom-up strategy, spe- 
cific model knowledge about a real world scene and its objects is used to fuse 
more or less primitive objects to primitive objects at the same or at a higher 
level. 

In the case of a top-down strategy the model of an object contains all relevant 
information about its physical properties called features (e.g. its shape or its 
composition by components / structural information). Of particular interest is 
the information about which component can be detected by which sensor. 
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Using this knowledge about a possible object the data of a lower level can be 
thoroughly analysed to increase or decrease the confidence in the assumption 
made before. 



sans or 1 
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„ situation 
refl 



1 



Fig. 3. Multi-level fusion functional architecture 
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There are three options to design the general fusion structure. This is sequen- 
tial fusion, parallel fusion and a looped fusion structure. These are three basic 
structural mechanisms that are in real systems combined with each other: In 
a sequential processing approach the chain of data processing can be followed 
bottom up level by level performing fusion operations between data elements 
that come from the same sensor. Due to this strategy the degree of fusion as 
well as the degree of confidence is increased continuously. The second option 
- parallel processing chains - occurs, if several sensors are available and if par- 
allel alternative processing is performed for the same sensor. Then the degree 
of composition increases, because components delivered from different sen- 
sors are combined and the sensor specific confidences are accumulated. 
Looped fusion can be implemented if detected features are assigned to a fea- 
ture model and a back loop according to the degree of composition is initiated. 
Doing that it should be possible to assign additional features defined by the 
feature model to the object hypotheses. The use of feature models and the 
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mentioned methodology allows increasing certainty and confidence of object 
estimations by the perception process itself. If the quality of the interim 
results is not satisfactory the data can be passed through an adopted process- 
ing chain again to improve the results. The back loops are located between 
several levels. The repetition of certain processing steps can result in more 
accurate and stable results. 

The state of the whole system represents a certain scenario in the real world 
how it is seen by multiple sensors. This overall information is the content of 
the environmental model. The environment description is a specific output of 
the environment model. This description can be used by an application to 
process certain information of the 4D environment model. 



2.3 The Grid-based Fusion Approach 

The main idea of grid-based fusion is to do sensor data fusion in an occupancy 
grid. This occupancy grid is a regular discretisation (sampling) of the environ- 
ment in cells, where each cell contains the probability that the corresponding 
part of the environment is occupied. Sensor data fusion is done in a generic 
way in occupancy grid framework. This approach allows an important sensor 
flexibility and sensor independence. Sensor data fusion could be done at dif- 
ferent levels: at a feature level, in this case, a precise model of each sensor is 
needed to build the grid, and at a track level, in this case, a geometric descrip- 
tion of the processed data is needed. 

The resulting occupancy grid is actually a snapshot of the current environ- 
ment, i.e. an instantaneous view of the surrounding environment of the vehi- 
cle at a fixed frequency. In an occupancy grid, each cell contains the probabil- 
ity that there lies an obstacle. Advantages of the occupancy grid framework: 

► the framework is sensor independent 

► the framework could deal with raw data (low level fusion) and with pre- 
processed data (high level fusion) 

► fusion of sensor data in each cell of the grid 

► priors (i.e. the initial probability for each cell) according to the occu- 
pancy of the whole space could be integrated; For instance, if we are in 
a crowded environment, the prior of occupancy is very high and the 
rate of false alarm and missed detection is very high, and the fault tol- 
erance of the system is increased 
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Fig. 4. Detail of the grid-based fusion architecture on the track/high level 

In the case of high level fusion, the sensors return for instance velocity, of 
which a map of velocities is built as in an optical flow framework according to 
the discretisation of the occupancy grid. Thus in each cell, a probability distri- 
bution over a set of possible discretised velocities is built. If the sensors can 
return a classification of the obstacles, a map of category (such as car, truck, 
bicycle, pedestrian, none) is established and in each cell a probability distribu- 
tion over all the possible categories is set up. 




Fig. 5. 



Detail of the grid-based fusion architecture on the feature level 
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Based on the occupancy grid, we extract the objects in the grid: Moving obsta- 
cles are extracted by first identifying the moving area. We proceed by differ- 
encing two consecutive occupancy grids in time. Then extraction of features 
such as surface and velocity could be performed to classify the type of obsta- 
cle. In case of the possibility of using velocity or a category map the previous 
process is fused with these two maps to obtain the gravity centre of each 
object and a probability distribution over its possible category. 

Data association algorithms are used to update the different tracks with 
extracted tracks. To deal with unobservable tracks, we use a particular process 
for handling occlusions. To deal with the particular topology of the environ- 
ment: entry and exit areas, we use a particular process based upon the 
roadmap. 

Thanks to the occupancy grid, we extract unobservable areas due to occlusions 
(Occlusion management). For each object, three cases arise: 

► The object is seen by the sensors and then is associated with sensor 
measurement. 

► The object is not seen due to occlusion, the sensors measurements 
define occluded areas and in each occluded area there is a uniform prob- 
ability that the object lies. 

► The object is not seen due to miss detection. When association is made, 
a probability over the velocity and the category is updated thanks to 
prediction models for each category of road users. 

Thanks to the interpretation of the map, we are able to know areas where the 
objects could appear or disappear (entry and exit areas). This helps the robust- 
ness relative to false alarm and missed detection. 

Object list are managed as a list of tracks. Each track is tagged with a specific 
ID and a set of characteristics. A track is created when consistent sensor infor- 
mation makes the existence of an object certain. One deletes a track only if 
this track disappears of the sensor view in a disappearance area. We also man- 
age merge and split of objects and all other relevant problems that might 
appear. To perform the tracks update, tracking methods are used and a set of 
hypotheses of predicted position, velocity and category for each object is 
obtained. 

In the situation refinement, we propose to perform trajectory prediction for 
the objects present in the environment. This trajectory prediction is per- 
formed using the classical "learn predict" paradigm. In a first step, the past 
trajectories of the objects are collected and are clustered to define some class- 
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es of typical trajectories. In a second step, these classes are used to predict the 
future trajectory of an object present in the environment. 

2.4 The Track-Level Fusion Approach 

The track-based fusion within the object refinement layer is a distributed 
approach. It assumes that tracking is carried out inside each individual sensor 
or system, and the tracks feed the track level fusion algorithms. It can be 
applied to automotive sensor networks with complementary or/and redundant 
field of view. The advantage of the approach is that it ensures system modu- 
larity and allows benchmarking, as it does not allow feedbacks and loops 
inside the processing. 

Research and development for track level fusion is focused on developing inno- 
vative algorithms in the area of multidimensional (N-D) track- to-track associa- 
tion, track management and track fusion. Expected results from these efforts 
are more consistency and the avoidance of spurious or invalid perception 
information. The output of the track level fusion is aggregated tracks in the 
union of the sensor field of views. 



Fig. 6. Track-based fusion functional architecture 

The research and development process for situation analysis consists of two 
main components: The first step is to develop the appropriate level of domain 
specific knowledge for the road elements (e.g. road borders, lanes, obstacles) 
and the second to develop a decision making process that is able to codify and 
manipulate the knowledge mentioned above. In situation refinement the sys- 
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tem is aware not only of the states of the road elements but has also knowl- 
edge of their relationships. The outcome of situation refinement enriches the 
environment model including additional attributes of the ego-vehicle and the 
obstacles (predicted paths, object to lane assignment, evidence for vehicle 
manoeuvres, etc.) 

The track level fusion architectural modules are depicted in the Fig. 6. It is 
implied that a set of track arrays are entering the fusion system while the out- 
put of object refinement process is consisted of the fusion object list. The inter- 
nal functionalities in this architecture are the association (spatial track assign- 
ment and 2-D and N-D association), the track to track update (fusion) and the 
fused object management. All these sub-modules are described in detail in this 
section. 

The Fusion Area Track Assignment is the first function that is imposed to the 
tracks when they are entering the fusion system. The main objective of this is 
to decrease the computational load of the overall procedure and also to ensure 
the configurability and the interoperability of the procedure. A set of sensor 
configuration parameters are necessary for this module to work properly. At 
least it is required to have the sensors' maximum range. Field Of View (FOV), 
direction, location, accuracy resolution and a performance index - estimation 
error covariance matrix or overall confidence level. The main process of this 
module is to separate the sensor coverage area around ego vehicle and conse- 
quently to separate the available tracks. These areas could be blind areas not 
observed by any sensors, areas with one sensor and areas observed by two or 
more sensors. The main result of this process is to divide the fusion problem 
to a number of smaller fusion sub-problems. 

The tracks that belong to areas without or single sensor surveillance are pass- 
ing to the output without any additional processing. On the other hand, the 
tracks that are within the common sensors areas (2 sensors or more) an asso- 
ciation measure will be defined. This metric is for generating the hypotheses 
for association between tracks, and then the relative association matrix or 
other metric passes to the next level where the track to track assignment takes 
place (Track to Track Association). In the case of 2 sensors tracks the 2-D asso- 
ciation problem is solved. The input to this module is the output of track to 
track association and its output is the pairs of tracks that are suitable for fusion 
and the not assigned tracks that simply pass to the next module. In the case 
of tracks coming from more than 2 sensors then the solution to this problem 
the N-D with N to be 3 more takes place in this module. Usually this problem 
concerns the sequential generation of a 2-D problem out of the N-D and after 
that the solution is similar to this acquired in 2-D assignment. 
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The assignment tracks (2 or more) that come from the output of the assign- 
ment modules are fused by the track fusion module. They are updated and 
generate a fused object state and covariance that replaces the existing sensor 
level tracks. 

Within the object management module the fused and the non-fused tracked 
objects are formatting the final object list output for the object refinement 
process. All the objects have an ID and in this module the initialisation, 
updates, deletion of objects based on ID information take place. Moreover, this 
module will handle, in a final step, object management issues such as duplica- 
tions of objects, blind areas objects, transition of objects between different 
areas and all other relevant problems that might appear. 



Fig. 7. The situation refinement internal functional module architecture 

The model collection, containing also motion models, is a horizontal activity 
and not only internal in object refinement. It is a function that generates and 
uses the available dynamics models, static models, sensor models and the rel- 
ative to the environment description models. This module is also necessary to 
the situation refinement process. 

The internal situation refinement modules are depicted in the Fig. 7. The out- 
put of the object refinement process is the main input in this module. The 
internal functionalities that take place in this architecture are the assignment 
to objects in a specific lane and the prediction of the path of the ego and the 
other vehicles (moving objects). The final output is the ego-vehicle and moving 
objects manoeuvre classification together with a confidence index. The output 
of this module passes to the application. 
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The object path module and the ego path module concern the prediction of the 
expected future path of moving vehicles and the own car respectively for a 
short period of time (e.g. 4s). 

The lane assignment module uses object position information and the available 
lane geometry and assigns a lane index to each of the vehicles accompanied 
with a confidence index. This information is very useful for the manoeuvre 
classification modules. 

The objects and ego manoeuvre classification modules analyse the behaviour 
of objects and the ego vehicle and classifies them according to a predefined 
discrete set of classes (e.g. overtaking, exceeding speed, parallel to the lane, 
lane change, etc.). These modules assume the existence of an environment 
model - i.e. descriptions of the road attributes and the lane properties and the 
output of the objects' path and the lane assignment; they analyze relationships 
between "objects" and produce a new structure. Their output will be part of 
the environment model and will be defined in the relevant tasks. The decision 
is accompanied with a level of confidence. 



2.5 The Fusion Feedback Approach 

To overcome a limitation of track-based fusion, some approaches insist on 
injecting information coming from other sensors at an early stage of process- 
ing by a given sensor. This can be achieved by feeding unprocessed - or slight- 
ly processed data - from different sensors into a single module in charge of all 
processing across different sensors. 

An original alternative way put forward here relies on fusion feedback. In 
track-based fusion architectures, taken as reference architectures, the output 
of fusion reflects information coming from different sensors. By feeding this 
processed multi-sensorial information into early processing of a given sensor 
(here sensor 1), we make it possible to sensor 1 processing to confirm some 
detection formerly only suspected by sensor 2. 

As an example, a RADAR device and a FIR camera originally produce tracks. 
Track-based fusion will merge some of them, and logically leave some RADAR 
and FIR tracks unmatched, because of possible complex and/or misleading 
environment. With Fusion Feedback, the RADAR information available on out- 
put from Track-Level Fusion will allow FIR processing to further rescan the 
area corresponding to the RADAR-only tracks. 
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Fusion Feedback builds upon a track-based fusion architecture, providing a 
solution to overcome its limitations through little structural changes. To do so, 
we want the processing of some given sensor - sensor 1 - to be able to use 
some additional information including data from sensor 2. The output is an 
excellent candidate, as it already exists in a traditional track-based fusion 
architecture. It contains information provided by both sensors. 



Therefore, we feed the output of track-based fusion into sensor 1 processing, 
as illustrated by the Fig. 8. Here, information coming from sensor 2 goes to 
track-based fusion, where some sensor 2 tracks might be left unmatched for 
various reasons. However, sensor 1 processing will use this information to bet- 
ter focus its search. This concept provides improved performance, without 
major reshuffling of the track-based fusion architecture, as it merely requires 
one additional link from fusion to the sensor processing, and customisation of 
the sensor processing. 




Fig. 8. Global Fusion Feedback architecture 

According to the addition of the link illustrated in the Fig. 8, the sensor 1 pro- 
cessing must be able to take advantage of information coming from track- 
based fusion. Considering a FIR camera as sensor 1 the original sensor 1 pro- 
cessing can be described as acquisition, pre-processing, detection, tracking 
and output formatting. Now, to accommodate the use of extraneous informa- 
tion, we have to enhance this architecture with additional sub-modules, and 
some enhanced existing sub-modules (cf. Fig. 9). 
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The information coming from the fusion module has to be converted into the 
natural representation for the sensor, here the FIR camera. Then detection 
must take into account the hint provided, possibly by releasing detection 
thresholds around given areas. Beside its usual functionality, tracking will now 
additionally have to check the compatibility between external track continuity 
and possible new FIR tracks detected using this hint. 



3 Conclusion 

In this paper we presented a modular framework for multi sensor data fusion, 
which addresses the crucial task of automotive environment perception. 
Thereby this interoperable architecture is not dependant on a special safety 
application but is designed on top of a multi sensor platform to serve diverse 
safety functions at once. Furthermore it will offer a robust and reliable per- 
ception performance due to the integration of diverse fusion approaches and 
algorithms for situation refinement beyond current state-of-the-art. 
Additionally this architecture with its extended environmental modelling pro- 
vides an excellent basis for further enhancement in the field of automotive 
environment perception and thus contributes to the design and development 
of road safety systems. 
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Framework Program. The long term goal of the subproject ProFusion2 is to 
streamline the sensor data fusion inside PReVENT by, e.g. gathering require- 
ments, defining certain standards and developing fusion algorithms, which 
should be used by the application-oriented subprojects within PReVENT. 
Hence ProFusion2, whose consortium consists of CRF, Daimler Chrysler, Delphi, 
FORWISS, IBEO, ICCS, INRIA, SAGEM, the University of Chemnitz and Volvo 
Tec, addresses research work of common interest related to sensors and sen- 
sor data fusion including a modular architecture for sensor data fusion. This 
architecture will be based on an extended environment model and represen- 
tation, consisting of a set of common data structures for sensor, object and sit- 
uation refinement data and algorithms as well as the corresponding models. 
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Abstract 

Automotive safety functions are divided, on one hand, into passive 
and active safety according to a time factor before an accident or a 
critical situation could occur; on the other hand, they focus on single 
slices around the intelligent vehicle, even if more and more sensors 
are added to improve system accuracy The paper proposes new 
integrated functions under INSAFES sub-project of PReVENT merg- 
ing and harmonizing lateral and longitudinal support functions, col- 
lision mitigation protective functions and road users' protection sys- 
tems. The advantage of INSAFES that is shown in the paper is that 
all functions share the same system resources - sensors and per- 
ception units - and trigger a given set of actuators and HMI con- 
trollers in order to communicate with the driver or intervene. The 
proposed system will be integrated and demonstrated in a passen- 
ger car and a truck. 



1 Introduction 

Existing sensing systems can identify the presence of obstacles and are able to 
support the driver, generally, in front of the vehicle or only on "single slices" 
of the scenarios around the vehicle. Consequently, different sensor technolo- 
gies need to be integrated today in the vehicle in order to cover all the area 
around it and to provide a complete support in all traffic and environmental 
scenarios. The proposed work intends to allow the extension of the operative 
scenarios of the advanced driver assistance and protective systems beyond 
their current limits and allows enabling a considerable step forward of the 
actual development and implementation of data fusion techniques. 

This problem is tackled in the INSAFES sub-project. The goal is to improve the 
functionality and reliability of applications developed within PReVENT 
Integrated Project (IP) and to advance from stand-alone safety applications. 
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targeting one specific function each, to an integrated system, covering a vast 
range of applications. This priority of INSAFES focuses on the full coverage of 
the surrounding of a vehicle, in order to warn the driver, intervene, or mitigate 
the effects of an accident. INSAFES addresses the proper use and interpreta- 
tion of all the information available from sensors or functions that are being 
developed in PReVENT [1], and the subsequent situation and risk assessment. 
The paper studies INSAFES use cases such as: 

a) enhanced longitudinal control functions (e.g. integration of frontal colli- 
sion warning and collision mitigation strategies, semi-autonomous brak- 
ing intervention: braking action confirmed by the driver on the brake 
pedal after a warning; warning and intervention strategies optimised 
according to the data received from communication components). 

b) enhanced lateral control functions (e.g. improvement of lane keeping 
support, optimised also with respect to the presence of lateral obstacles 
e.g. close guard-rails or parallel vehicles; side collision avoidance/miti- 
gation for trucks within the ego-lane, typically in drifting situations) 

c) integration and synergies of lateral and longitudinal control applica- 
tions (e.g. all-around warning function, based on the reconstruction and 
the assessment of all-around traffic scenario; improved frontal collision 
warning/mitigation: optimisation of the warning/intervention strate- 
gies, based on computation of a feasible evasive trajectory; collision mit- 
igation, autonomous intervention: optimisation of the intervention 
strategies - for instance switch to semi-autonomous modality - depend- 
ing on the detected surrounding traffic). 

The developed functions will be implemented, demonstrated and evaluated 
through validation and functional tests using a passenger vehicle from FIAT 
Research Centre and a truck from Volvo Technology. The equipped vehicles 
serve for test and validation purposes and will be used for testing the func- 
tionality and performance of the developed system and for evaluation of the 
driver acceptance of the human-machine-interface. INSAFES belongs to the 
cross-functional activities of PReVENT Integrated Project, which in turn is co- 
funded by the European Commission under the 6 th Framework Program. The 
project is coordinated by Volvo Technology; it started in September 2005 and 
will finish in December 2007. 

The paper starts with a layered conceptual framework borrowed from a 
PReVENT cross-functional activity, ProFusion2, which deals with sensor 
fusion, and implemented to cover INSAFES needs. A chapter that investigates 
how existing active and preventive safety functions can be improved in 
IN SAFES follows. In Chapter 4, eight new functions are briefly presented 
which integrate different functional fields and are proposed for future imple- 
mentation in the INSAFES passenger car and truck. The functions are classi- 
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fied with respect to the type of vehicle, but it can be considered without loss 
of generality, as integrated functions for future vehicles. 



2 INSAFES System Design Layers 

Due to the complexity of the information handled in INSAFES, the high quan- 
tity of information sources and the system requirements a sensor fusion 
approach is adopted. The same approach can be met in other sub-projects of 
PReVENT and is also recommended from the ProFusion PReVENT cross-func- 
tional activities [2]. ProFusion's major objective is to push the sensor data 
fusion techniques used for automotive environment perception beyond the 
current state-of-the-art. This is done by setting up a modular and interopera- 
ble fusion architecture for multi sensor systems, which integrates diverse 
approaches (e.g. low and high sensor data fusion, algorithms for situation 
refinement, etc.). In doing so, the ProFusion framework is not limited to any 
special safety function but operational with all different kinds of applications 
like collision mitigation, lateral control, lane departure warning and others, 
and thus is suitable for INSAFES. 




Fig. 1. Functional layers of a sensor fusion active/preventive safety sys- 
tem, the INSAFES paradigm 



In general, sensor data fusion includes several processing steps which take the 
data of several single sensors and combine the information from them in order 
to achieve a certainly better result than the outcome of each single sensor's 
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processing could provide. This effect of getting better or more adequate results 
out of sensor data fusion compared to single sensor processing is called 'syn- 
ergy' or 'synergetic effect'. A typical driver assistance system's architecture 
consists of at least three layers: the perception layer being responsible for the 
environment perception, the decision/application and the action layer, which 
are responsible for taking and executing actions respectively. Sensor data 
fusion, may take place on every individual layer depending on the approach or 
even in all layers. Usually, by sensor fusion, one can assume fusion in the per- 
ception layer (e.g. for obstacle detection), but this is not true, especially in the 
case of INSAFES project. However, in the system's perspective this is indiffer- 
ent, if the layers support the same input output interfaces. 

The INSAFES project develops the common decision layer and study the inter- 
facing to it from the different functions. One challenge is be to find a level of 
integration that result in an increased performance but still enables most of 
the different function components to be re-used. The INSAFES idea is to inte- 
grate already existing function components, not reinventing them from 
scratch again. 

The action- and output-coordination will be covered, in close cooperation with 
the AIDE project [3]. The idea of introducing a common decision layer is 
depicted in the figure above; an action coordinator still is needed if there are 
functions that are not fully integrated in the common decision concept and it 
is described in AIDE related publications [4]. 



3 PReVENT Functions as a Starting Point of INSAFES 

To clarify how IN SAFES gives added value to the respective stand-alone func- 
tions - represented by PReVENT vertical sub-projects - the main goals of each 
of these projects are outlined here, together with a short description of the 
advantages of the INSAFES approach. This will supplement the description of 
the complementary character as well as the synergies between the projects 
highlighted throughout this paper. 

The APALACI functions serve two principal goals: (1) the development of 
advanced obstacle detection functionalities, and (2) the development of longi- 
tudinal pre-crash and collision mitigation applications. Both functionalities are 
based on sensor fusion techniques in order to be able to detect and to classify 
objects including pedestrians. 
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Main INSAFES added value: Integration with the front collision warning func- 
tionality for the optimisation of the semi or autonomous braking intervention 
strategies with respect to the complete traffic scenario reconstruction; exten- 
sion of irreversible pre-crash function to the lateral area (early detection of lat- 
eral obstacles for lateral air-bag pre-set). 

The target functions for LATERAL SAFE are aiming to develop (1) a lateral and 
rear area monitoring application enhancing the driver's perception of risk of 
collision in the lateral and rear area of the vehicle. (2) a stand-alone lane 
change assistance system with integrated blind spot detection assisting the 
driver in lane change manoeuvres while driving on roads with more than one 
lane per direction and (3) a lateral collision warning function that detect and 
tracks obstacles in the lateral field and warn the driver about an imminent risk 
of accidents (collision, road departure or merging situations). 

Main INSAFES added value: All-around collision warning function, integration 
of lateral support functions; reconstruction of the surrounding scenario for 
optimising the longitudinal functions and control strategies for warning or 
mitigation (e.g. strategies improved according to the computation of feasible 
evasive manoeuvre). 

The SASPENCE concept aims at supporting the driver in avoiding accident sit- 
uations related to inappropriate speed and/or insufficient distance (headway) 
to the obstacle ahead, by providing information through tactile primary com- 
mands. The system should support in maintaining an appropriate vehicle 
velocity (and distance) for any given driving situation in any conditions. Using 
such a system as starting point and collecting information from "all-around 
sensors", as well as complete data-fusion algorithms, more advanced informa- 
tion can be provided to the driver, concerning the suggestion of the optimal 
manoeuvre to perform, being a braking or obstacle avoidance. 

Main INSAFES added value: Integration of longitudinal control functions; 
improved scenario reconstruction and warning strategies using WILLWARN 
information. 

The SAFELANE application aims at enhancing traffic safety by supporting pri- 
marily the drivers of commercial vehicles in avoiding unintended lane or road 
departures on motorways and rural roads. The system operates safely and reli- 
ably in a wide range of even difficult road and driving situations. If an unin- 
tended lane departure is foreseen or detected, the driver is warned or an 
active steering system is activated to keep the vehicle within the lane or to 
bring it back into it. 
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Main INSAFES added value: Integration of lateral control functions 

A basic WILLWARN system will focus on warning of reduced visibility or 
reduced friction or if the car has failed and has become an obstacle for other 
cars. Obstacle information will be derived from an ACC sensor and speed pro- 
files or avoidance manoeuvres will be recommended. As IN SAFES is using a 
multiple sensor approach for obstacle detection, the system will be capable of 
detecting and classifying obstacles in high quality. An IN SAFES vehicle can 
warn other vehicles by car to car communication when an obstacle is detect- 
ed using the techniques and protocols that are developed by WILLWARN. 

Main INSAFES added value: integration of longitudinal control functions, radar 
systems integrated to WILLWARN functions. 

The following highlights the complementary character of the project propos- 
als by sensor technology and actuators. 
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Fig. 2. Sharing of resources for the realization of integrated functions 



4 INSAFES Functions 

4.1 Adaptive Lane keeping Support with Lateral Monitoring for Trucks 

Lane keeping support function (represented in PReVENT as SAFELANE) 
assumes the centre position of the ego- lane as the optimal driving position. In 



Integrating Lateral and Longitudinal Active and Preventive Safety Functions I 477 



case of presence of a vehicle in the adjacent lane and especially if lanes are 
narrow and thus, lateral distance between vehicles is small, a lateral displace- 
ment of the own vehicle in the own lane is desirable to increase the lateral dis- 
tance. In this function, the lane keeping algorithm is modified to consider a dif- 
ferent desired lane position, when close vehicles in adjacent lanes are present, 
as shown in the following figure. The function may have different levels, start- 
ing from just modifying parameters in the lane keeping support algorithm up 
to actively correcting the vehicle's position in the lane by using a steering 
actuator. The existence of an adjacent vehicle will be presented to the driver 
from a different application. 
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Fig. 3. INSAFES interactions with existing functions and cross-functional 
activities 

The goal of the function is the enhancement of driving safety by adapting the 
desired lane offset for the Lane Keeping Support with respect to distance of 
other vehicles in adjacent lanes. The goal is to avoid safety critical driving sit- 
uations by monitoring the lateral distance to vehicles driving in adjacent lanes 
and in case the distance is too small and thus implies an increased risk for 
accidents to adapt the desired lateral offset and position in the own lane of the 
lane keeping algorithm. 
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Fig. 4. Enhanced lane keeping with adaptive lane offset 



4.2 Lane Change Warning with Haptic Steering Wheel Feedback and corrective 
Action for Trucks 

The objective of this function is the enhancement of driving safety by com- 
bining an originally passive lane change warning function (cf. LATERAL SAFE) 
using only e.g. optical and/or audible warning with the possibilities provided 
by using SAFELANE's active steering actuator for providing haptic feedback 
via the steering wheel. The goal is to avoid safety critical driving situations by 
monitoring the lateral area of the ego-vehicle and in case of an indicated lane 
change (e.g. usage of blinkers and actively steering) while another vehicle is 
present in the lateral area to issue a warning by haptic feedback (e.g. vibration 
and corrective action) by using the active steering actuator from SAFELANE. 




Fig. 5. Lane change warning with haptic feedback and correction 



The lane change warning function uses visual and/or audible warnings. By 
alternatively or additionally using the SAFELANE active steering actuator a 
different HMI channel, a haptic warning, can be used. Furthermore, not only 
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a warning can be given, but by applying an overlay torque to the steering 
wheel a correction of the steering angle and "prevention" of lane change 
maneuver can be achieved as well, as illustrated in Fig. 5. 



4.3 Enhancement of APALACI low Speed Obstacle Warning for Trucks 

The objective is the enhancement of APALACI low speed obstacle warning and 
right- turn start inhibit with a blind spot sensor (right). This is a beneficial func- 
tion as it covers a remaining blind spot between the APALACI front sensors 
and the lateral monitoring sensors used for the previously described functions. 

The APALACI realization in the Volvo truck focuses mainly on the start-inhib- 
it application, aiming at preventing run-over accidents in start-up situations, 
e.g. accidents when an object or a pedestrian in the proximity of the vehicle is 
hit by the truck taking off from standstill. Since the start-inhibit function 
requires the system to perceive the area close in front of the vehicle, it was 
also considered possible to extend the functionality to a general proximity- 
warning function: warning if there is an obstacle detected within the specified 
area, when the truck drives at low speed. The area of detection which has to 
be covered by the sensing system is shown in the figure below. The area for 
proximity warning is larger than the start inhibit area. In particular the right 
side of the vehicle needs also to be covered in order to include the main blind 
spots. This was not included in APALACI, but is considered here to enhance 
the APALACI function with the low speed obstacle warning, including espe- 
cially the blind spot area as indicated with the red ellipse in Fig. 6. 




Fig. 6. Enhancement of APALACI low speed obstacle warning 
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4.4 Enhanced Curve Speed Warning Function for Trucks 

The curve speed warning function is based on reliable perception of the road 
ahead. The road and lane curvature can be derived by a lane tracking system 
from SAFELANE - which is accurate for short distances - but still digital map 
data are needed for a longer electronic horizon and will be provided by 
MAPS&ADAS project. Considering the general condition of the truck, trailer, 
load weight and type (if known) or environmental conditions (if known) a suit- 
able lateral acceleration and thus a suitable velocity for driving into and 
through the curve can be calculated and can e.g. be given as recommendation 
to the driver or in a second stage issue a warning. Finally even an active inter- 
vention by actively limiting or reducing the velocity is a possibility, but not 
considered within the scope of the project. 




Fig. 7. Enhanced curve speed warning for trucks 



4.5 Forward-Lateral-Rear-Monitoring Function for Trucks 

The Forward-Lateral-Rear-Monitoring (FLRM) function is based on the Lateral- 
Rear-Monitoring (LRM) function developed for the simulator in LATERAL SAFE 
project. By using the lateral sensors, the additional blind spot sensor (right) 
and furthermore including the forward area obstacles information from the 
APALACI sensors the Monitoring functionality will be brought from simulation 
to a real application. As the right side blind spot is not visual in the original 
APALACI display, the FLRM display and visualization serves for the APALACI 
low speed obstacle warning functionality as well. 

The overall objective of the FLRM function is to inform the driver of adjacent 
vehicles. This will assist the driver in the task of detecting and tracking sur- 
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rounding vehicles. FLRM thus extends and improves the visibility in direct 
(window openings) and indirect (mirrors) in the following ways: 

Using multiple sensors will cover all of the vehicle surroundings, effectively 
eliminating the blind spots normally found in mirror systems. 

The arbitrary placement of a sensor-display based system does not require an 
optical line-of-sight. This is especially valuable for trucks where ego 
vehicle/trailer typically blocks rear line of sight and where a significant blind 
spot area is to the right side of the cabin and even directly in front of the truck, 
as covered in APALACI. 

The integration of sensors presented in a single consistent HMI will eliminate 
the fragmented viewing; all of the vehicle can be surveyed from a single point. 

The nature of the FLRM function is thus passive, the function will present 
information; it is up to the driver to choose if and when to make use of the sys- 
tem and the information presented. The FLRM application presents the infor- 
mation in an abstract per-sector level and basically informs the driver of the 
presence of a vehicle in a designated sector around the vehicle as shown in 
Figure below. 





Fig. 8. Forward-Lateral-Rear-Monitoring Coverage 



4.6 Integrated Longitudinal Support in Passenger Cars 

This function results from a combination of functions developed inside WILL- 
WARN, SASPENCE, and APALACI subprojects as it merges preventive and 
active safety systems in the longitudinal driving axis. It provides support and 
protection in cases that an accident may occur in a few milliseconds or a crit- 
ical situation or a hazard is foreseen for the future. 





482 I PReVENT 



The driver is invited to follow a speed profile that avoids entering into dan- 
gerous situations with excessive speed (safe speed). This suggestion can be 
communicated to the driver through different perceptual channels, that is: 
visual display, sounds, haptic feedback on the accelerator pedal, and vibration 
on the seat-belts. When the driver pushes the brake pedal, but the force 
applied by the driver is not sufficient to avoid a collision, extra pressure is 
applied by the system on the brake (assisted braking). Finally, just before 
entering into a collision situation, safety belts are pretensioned to mitigate the 
effects of the accident. 



4.7 All Around Collision Warning in Passenger Vehicles 

The All-around Collision Warning function should warn the driver in case of a 
likely accident due to approaching objects from all sides. This includes traffic 
coming to close from behind, oncoming traffic from the side, and also oncom- 
ing traffic in the front useful e.g. in the case of driver distraction. 

A network of sensors all-around the ego-vehicle is mandatory to sense the sur- 
rounding area. A threat assessment is calculated for all detected objects and a 
pruning algorithm selects the critical ones. Based on this, in case of a danger- 
ously approaching object, a warning is given to the driver early enough to 
allow time for a situation analysis and a reaction. 

To separate this application from other safety applications like pre-crash or col- 
lision mitigation, the focus has to be set to different main areas. Due to the fact 
that too many warning issues would annoy or overstrain the driver instead of 
leading to a real safety benefit, the target scenarios have to be limited by 
reducing the relative speed and operative speed intervals when the function is 
active. Most accidents occur in the urban scenarios which should therefore be 
the main scenario covered by the application. Another important argument for 
the limitation of the relative speed interval is the finite reaction time needed 
for the driver to recognize the situation and react in accordance to that. Due 
to the early warning function, it is intended that it is possible to reduce the 
number of accidents especially in the case of driver distraction or, at least, to 
reduce the crash severity by the reaction of the driver. 

Warning information has to be given to the car driver by an unambiguous and 
appropriate readout. To allow the driver to understand the situation immedi- 
ately this information has to include the direction/area of the critical situation 
in an intuitive way. Further investigations about an optimal solution for this 
task are clearly recommended. 
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4.8 Maneuver Suggestion Function 

The objective of the function is to reduce the number of accidents due to a 
wrong maneuver undertaken by the driver, in particular when a decision has 
to be taken between staying in the lane or change lane (overtake). The driver 
receives a visual suggestion about the fact the he should stay in the lane or 
change lane, considering the position of the car on the road and the presence 
of obstacles. It is an extension of SASPENCE safe speed suggestion function. 



5 Conclusions 

The paper presented a new approach of merging different active and preven- 
tive safety functions and proposed integrated functions under the framework 
of PReVENT function taxonomy. The principles and basic requirements were 
set at the beginning of the paper; however, the actual design and implemen- 
tation of the architecture is under development and results will be presented 
in the near future. 
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Lane Detection for a Situation Adaptive Lane keeping 
Support System, the SAFELANE System 



N.Mohler, D.John, M.Voigtlander, Fraunhofer IVI 



Abstract 

The goal of the SAFELANE project is to develop a situation-adaptive 
system for enhanced lane keeping support. A prerequisite for lane 
keeping is reliable information of the vehicle environment. 
Especially the vehicle position within the lane and the course of the 
road ahead is important. This information is provided by the lane 
detection component. The lane tracker is mainly vision based but it 
is also supplemented by map, positioning and vehicle data. A high 
dynamic range camera provides the processing unit with image 
data. Measurement points of lane borders, calculated by a robust 
edge detection algorithm, are used to estimate a 3D clothoid model 
of the lane. A new approach for getting an initial lane model is intro- 
duced which can be dynamically adapted from map and positioning 
data. The detection is extended to the neighbour lanes and a lane 
markings type classifying component is added. 



1 Introduction 

The SAFELANE project is part of the Integrated European project framework 
PReVENT. The project's objective is to develop a lane keeping support system, 
which should go beyond the currently available systems and should function 
safely and reliably in a wide range of even difficult road and driving condi- 
tions. It should be usable in different type of vehicles from small passenger 
cars to commercial trucks and be operable on motorways and rural roads 
throughout Europe. Additionally to passive lane keeping support with haptic, 
acoustic and optic HMI elements it is intended that the system actively steers 
back into the lane in critical situations. 

To reach that ambitious goal, reliable information of the vehicle environment 
is necessary. Especially the vehicle position in the driving lane and the course 
of the road ahead is important for lane keeping. Therefore, the lane detection 
system (lane tracker) is a key component of the overall system. Our lane detec- 
tion is mainly vision based but is also supplemented by map and positioning 
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data, vehicle and radar data in cases of missing lane markings, so as the sys- 
tem to be always available. 

This paper describes the specification and the current development status of 
our lane tracker. A special high dynamic range camera serves as input data 
source for the image processing lane tracking algorithm to cope with various 
lighting conditions. The image processing is done in real time on a common PC 
hardware platform. Measurement points of marked but also unmarked lane 
borders are calculated via a robust edge detection algorithm. The measured 
border points are projected onto the road plane and a lane model (clothoid) is 
estimated. To be as flexible as possible and to get accurate estimation results 
for the automated steering modus, a clear separation is made between camera 
calibration parameters and the estimated vehicle position parameters. 

To enhance the system performance and robustness additional information 
about the upcoming road segment like lane width and curvature is taken from 
a digital map and several algorithm parameters are adapted accordingly. A 
lane marking type classification and a neighbour lane detection are introduced 
to get more information about the current traffic situation. 



2 System Components 

The central component of the vision system is the main lane recognition. If the 
main lane recognition was successful neighbour lanes are trying to be recog- 
nized near to the main lane. Measurement points which are generated during 
the lane recognition process serve as input data to the lane marking type clas- 
sification. The obstacle area detection marks regions in the image where other 
objects are located like a vehicle driving ahead. The lane recognition uses this 
information to restrict the search of measurement points to free road areas. 
For the lane recognition as well as for the obstacle area detection the vehicle 
speed is needed. Map data are used to adapt certain lane recognition algorithm 
parameters like the current lane curvature and width. 



3 System Requirements and Specification 

The lane detection development and innovation were driven by the evaluation 
of existing state of the art lane tracker systems. Three different lane trackers 
were tested during the project concept and specification phase. Several criti- 
cal scenarios were identified, where the trackers produced false results or 



Lane Detection for a Situation Adaptive Lane keeping Support System, the SAFELANE System I 487 



failed. The found scenarios were investigated to get new ideas and approach- 
es for improving the performance of the current lane tracking technology. 
Some improvements and extensions came from lane keeping functional 
requirements (e.g. lane marking type recognition). 




Fig. 1 . System components overview 



4 Main Lane Recognition 

A model based approach with the lane as a clothoid was chosen for the lane 
recognition. Because highways, country roads and also rail tracks are built 
after a clothoid geometry a clothoid suggest itself as lane recognition model. 
While some systems model the lane borders in the 2 dimensional image coor- 
dinates, Dickmanns et al. [1] was the first who showed the benefits of using a 
3 dimensional spatial clothoid model for the lane recognition task. The road 
surface is assumed to be flat and therefore it can be modelled as a plane. The 
spatial vehicle state is described by its position and orientation relative to the 
lane. A model state vector comprises of the following parameters: lane width 
b, lane curvature c 0 , lane curvature change c lr lateral offset of vehicle relative 
to lane centreline y 0 , pitch angle relative to road plane cp z , yaw angle relative 
to lane tangent cp 3 

In images taken from the camera search areas are set depending on the mode 
the system is currently in. Measurement points are searched on lines in the 
image plane. At program start or if the system "lost" the lane, no valid lane 
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state is at hand - the system is in initialisation mode. Because there are no spa- 
tial hints about the lane ahead available, the search area extends almost to the 
whole image on rigid wide lines (scan lines) in initialisation mode. If the lane 
recognition was successful in the previous image frame, a valid lane state 
exists and can be projected onto the image plane. Because the lane state frame 
to frame differences are usually small it is reasonable to set the scan lines 
along the lane projection. In both modes a list of scan lines serves as basis to 
search for lane contour edges in the measurement algorithm step. Now one 
edge per scan line has to be selected out of the calculated edges being most 
likely belonging to a lane border. Finally the selected edges are fed to the esti- 
mation stage where a new lane state is calculated. 



Data source Search area Measurement Selection Estimation 




Image data 



List of 
Scan lines 



List of potential List of selected 

edges edges 



Lane state 
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Fig. 2. Lane recognition processing steps, pictures above taken in initiali- 
sation mode and below in tracking mode 



4.1 Camera Model 

To be as flexible as possible concerning camera and lens hardware an extend- 
able modular camera model is used. A camera model does the transformation 
from world coordinate points to image coordinate points (pixels). The reverse 
transformation from pixels to world points is a 2 to 3 dimensional transforma- 
tion and needs an additional condition like world points belonging to a known 
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plane or having a certain distance from the camera in common. The model 
comprises 4 transformation stages: Sensor chip model. Distortion model. 
Projection model. Rotation and translation model. Sensor chip-, distortion- and 
projection parameter are usually called intrinsic camera parameters and rota- 
tion and translation parameters extrinsic camera parameters. 
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Fig. 3. Pixel to world coordinate transformation 



The sensor chip model calculates an (may be distorted) image plane point 
(image plane: see below) out of a pixel position on the sensor chip. Given an 
image plane point the distortion model describes its distortion. There may be 
no distortion at all (pinhole camera) or e.g. radial, tangential, shearing distor- 
tion. Currently a radial distortion model is implemented. The projection model 
projects a camera coordinate point onto the undistorted image surface like a 
plane or a sphere. Models for central projection or central sphere projection 
are currently provided. The camera rotation and translation model describes 
the camera position and orientation in some world coordinate system. These 
systems are governed by a rotation matrix and a translation vector. 



Image coordinate system 




Fig. 4. Depiction of camera to world transformation 



To obtain the calibration parameters (intrinsic and extrinsic) of a camera instal- 
lation we use software for intrinsic and extrinsic calibration which generates 
calibration parameter files to be used directly in the vision system. It might be 
objected that this highly modularized approach creates "slow" code. But the 
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model is implemented with generic programming techniques (C++ tem- 
plates) so that the compiler has the chance to do all the optimisation it would 
do as if a monolithic approach would followed. With the described flexible 
camera model much higher accuracy of lane recognition results are achievable 
which the automatic steering actuator module of the SAFELANE system ben- 
efits from. Above that it is possible to use wide aperture angle optics (radial 
distortion and sphere projection) to widen the field of view. 



4.2 Lane Model 

In lane tracking systems usually the position and orientation parameters (lat- 
eral offset, yaw and pitch angle) to be estimated are preset with calibration 
values and the rotational calculation is simplified (sin(x) = x, cos(x) = 1). It is 
assumed that the camera is mounted in the vehicle only with a small pitch, 
yaw, and roll angle. 

One requirement of the SAFELANE system is that it should be able to operate 
on a variety of different vehicle platforms from small passenger cars to huge 
trucks. In a passenger car the simplification is justified. But in a truck with a 
high camera mounting position and relatively large calibration pitch angle, the 
assumption is violated. Therefore a clear separation is made between extrinsic 
camera parameters and vehicle position and orientation parameters to be esti- 
mated. This is done by dividing the world coordinate system into a vehicle- 
and a lane-coordinate system. Image points are first projected onto the ground 
plane of the vehicle coordinate system (XY-plane) with the camera model 
(intrinsic and extrinsic) and then transformed into the lane coordinate system 
in which the clothoid model is defined. The transformation comprises a rota- 
tional part (pitch angle cp z and yaw angle q) 3 ) and a translational part (lateral 
offset y 0 ): 
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where p l is a point in the lane coordinate system and p v a point in the vehicle 
coordinate system 

A clothoid is mathematically unhandy, so it is approximated by a third order 
polynomial: 
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2 u 6 



(2) 



where c 0 is the curvature and c 1 is the curvature change. 



A lane border measurement point can be calculated with: 




(3) 



where b is the lane width and + (-) correspond to a left (right) lane border. 



lane coordinate system 

Fig. 5. Depiction of vehicle and lane coordinate system 

4.3 Search Area 

In initialisation mode the scan line generation is quite simple and follows a 
rigid setting scheme. In tracking mode scan lines are set along the projected 
predicted lane state. A scan line is set perpendicular to the local lane border 
tangent (not necessarily horizontal) in word coordinates. The line width and 
space to the next line are calculated depending on the look ahead distance. If 
the current lane border is classified as marked, the average lane marking 
width is calculated and will be added to the scan line width. Projected in the 
image plane the slope of the scan line is checked and the scan line is rejected 
if a certain threshold is exceeded. 



vehicle coordinate system 





lane model (marking) 
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4.4 Measurement Points 

Fig. 6 depicts the process of generating measurement points. In a first stage it 
is sometimes helpful or even necessary to filter image data at pixel level. There 
are three mayor causes that can disturb the lane border measurement gener- 
ating process: At dark lighting conditions cameras tend to get noisy the road 
pavement structure can be coarse and heterogeneous and lane markings can 
be partially rubbed off. In these situations a filter is needed which on the one 
hand has smoothing characteristics but on the other hand retains strong 
edges. The median filter satisfies these needs. The median of a set of values is 
the middle value of the sorted set. In the image processing domain a median 
is calculated in a square area (called image filter mask) with a certain mask 
width. The median filter causes two problems: It is a computational expensive 
operator because of the involved sorting and structures disappear if smaller 
than the filter mask. We tackled the first problem by using an intermediate 
image, where already calculated median values are "cached". This makes sense 
because in the following processing stages edge and structur tensor computa- 
tion as well as segments calculation pixel are accessed several times. To pre- 
vent lane markings which appear small at further distances in the image from 
being smoothed away the filter mask width is adapted according the look 
ahead distance. 
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Fig. 6. Block diagram of measurement point generation 



The well known Sobel operator [2] is used for gradient calculation as basis for 
the edge detection. Edges are detected where the gradient magnitude is max- 
imal in gradient direction, a threshold is exceeded and the gradient points into 
the local direction of the lane. At the detected edge pixels the structure sensor 
[2] is computed. The quality of an edge pixel is computed by the ratio of its 
eigenvalues and the angle between the greater eigenvector and the tangent 
on the projected lane border. 



If the quality of an edge is good enough a further validation is done by calcu- 
lating little edge segments. The Search area of a segment is a small rectangle 
with the edge pixel at focus in the rectangle center, aligned along the lane bor- 
der direction, bounded by the adjacent scan lines and with a width derived 
from the maximal allowed difference angle between the edge- and lane bor- 
der-direction. Within the area neighbour edges in both lane border directions 
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are tried to be followed repetitively until the search area boundary is reached. 
If the segment is long enough with respect to the search rectangle length the 
segment is considered being part of a lane border. 




Fig. 7. Edge segments (red) at lane marking 



4.5 Selection 

In initialisation mode, where no lane state prediction of a previous time step is 
available an extended form of a Hough Transform [2] is employed. The Hough 
Transform is a technique which can be used to isolate features of a particular 
shape within an image. It requires that the desired features are specified in 
some parametric form. 

The Hough technique is particularly useful for computing a global description 
of a feature, given (possibly noisy) local measurements. The motivating idea 
behind the Hough technique (e.g. for line detection) is that each input meas- 
urement (e.g. image point) indicates its contribution to a globally consistent 
solution (e.g. the physical line which gave rise to that image point). 

As a simple example, consider the common problem of fitting a set of line seg- 
ments to a set of discrete image points (e.g. pixel locations output from an edge 
detector). We can analytically describe a line segment with the linear equation 
y = mx + b. For any point (x, y) on this line, m and b are constant. In an image 
analysis context, the coordinates of the edge segment points (x ir y { ) in the 
image are known and therefore serve as constants in the linear equation, while 
m and b are the unknown variables we seek. By plotting straight lines in an 
mb-coordinate system defined by each (x^ y { ), points in image coordinate space 
map to the so called Hough parameter space. If viewed in Hough parameter 
space, points which are collinear in the image space become readily apparent 
as they yield straight lines which intersect at a common (m, b) point. 
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The transform is implemented by quantizing the Hough parameter space into 
finite intervals (accumulator cells). As the algorithm runs, each ( x ir y±) is trans- 
formed into a discretized ( m , b) straight line and the accumulator cells which 
lie along this straight line in Hough space are incremented. Resulting peaks in 
the accumulator array represent strong evidence that a corresponding straight 
line exists in the image. 

Applied to the problem of lane measurement selection the Hough transform 
equation is defined by the clothoid model, the vehicle position and direction 
parameters. To reduce the complexity and satisfy runtime constraints, not all 
parameters are taken into account and the look ahead distance where meas- 
urement points are searched is taken back comparing to the tracking mode. 
The clothoid model is further simplified by omitting the lane curvature change 
term and a pitch angle variation is not considered. Hence the resulting Hough 
parameter space is 4 dimensional and comprises the parameters b (lane 
width), c 0 (lane curvature), y 0 (lateral offset) and q) 3 (yaw angle). The limits and 
the resolution of the Hough accumulator array are preset out of a configura- 
tion file. The parameter limits of c 0 and b are dynamically adapted through 
map data (if available). The algorithm takes each measurement point and 
stores a measurement reference in all accumulator cells satisfying the Hough 
transform equation. In a next step a number of accumulator cells are identified 
which accumulated most measurement points. To give more weight to meas- 
urement points of marked lane borders these points are counted double. To 
avoid initializing on guard rails or other lane marking like infrastructure the 
accumulator cell with smallest lane width parameter is chosen. All the accu- 
mulated measurement points in the chosen cell are used as input data for the 
following lane model estimation process. 

The Hough parameter of the chosen accumulator cell is a good lane model 
hypothesis but far too inaccurate as a final model result. Still they can be used 
as initial lane state parameters for the lane model estimation. 

Since the used estimator (Extended Kalman filter [3]) expects noisy measure- 
ments a variance value for each measurement has to be provided. The vari- 
ances of all selected measurement points are set according the distance to the 
lane state, the pixel quantization error and the edge type (marked / unmarked). 

During tracking mode the contour edge candidate next to the predicted lane 
state is used as a measurement point on a scan line. The variance of the select- 
ed measurement point is set the same way it is described for the initialisation 
mode. 
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4.6 Lane Model Estimation 
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Fig. 8. Lane model state estimator 

Assuming all measurement points are collected in a measurement vector x, 
the optimal state s opt can be estimated by minimization of the square error: 



V argmin (x- f(s)) T R~\x- f(s)) (s -s 0 ) T P~'(s -s 0 ) (4) 

The covariance matrices R and P of the measurement noise and the initial 
state s 0 , respectively are assumed to be diagonal. I.e. no correlations among 
measurement points and among initial state parameters are assumed. The 
variances of measurement points and fixed variances of the initial state are 
used for the estimation. 

Since the measurement function f is quite complex and nonlinear the estima- 
tion is done by linearization and iteration within an Extended Kalman filter [3]. 

To check the plausibility of the new state, tolerance intervals for each state 
parameter are given, which describe the admissible range of the lane model 
parameter. If one of the parameters lies outside of its interval, the estimation 
is rejected and the system is put into initialisation mode. 



5 Neighbour Lane Recognition 

In the concept and specification phase of the SAFELANE project it was identi- 
fied that the information of the presence of neighbour lanes can be helpful or 
even is necessary in critical traffic conditions for the lane keeping task. Thus 
the two directly neighboured lanes of the main lane where the vehicle is cur- 
rently driving in are estimated as well. This is done with the same methods 
used for the main lane besides some deviations in the initialisation mode. If the 
main lane recognition was successful the neighbour lane recognition is trig- 
gered. The initial search areas of neighbour lanes can be set more precisely 
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comparing to the main lane because we know the location of one neighbour 
lane border and usually neighbouring lanes have about the same width. The 
pitch angle of the neighbour lanes is assumed to be identical with the main 
lane. 

False detections can happen on structures that look like lane borders (e.g. 
crash barriers). Therefore two additional tests are applied. The first one com- 
pares the road texture of the main lane and the neighbour lane. If the texture 
difference exceeds a threshold the test fails. The second one checks if there 
are any obstacles inside the image area of the neighbour lane by means of the 
object mask image. If one of the tests is not successful the neighbour lane 
recognition will be rejected. 




Fig. 9. Neighbour lane recognition, road texture check in yellow windows 



6 Lane Marking Classification 

The knowledge of the road marking types which delimits the own lane is of 
great importance for the lane keeping task. The system should e.g. react more 
sensitive regarding lane departure in case of solid lane markers comparing to 
dashed markers. One information source for road marking types is the digital 
map. But on a multi lane road the GPS and map data are not accurate enough 
to locate the vehicle in a dedicated lane. 
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Preprocessing Feature extraction Filtering Classifier 




Lane mea. List of [SL, GL, MW] Feature vector Marking type 

points and variances [SL, GL, MW] 

and variances 



SL = Marking segment length 
GL = Marking gap length 
MW = Marking width 

Fig. 10. Block diagram of lane marking classification 

Lane markings are assumed to be solid or dashed marker lines. Dashed lane 
markings have certain constant marking line width, line segment length and 
line gap length. Therefore these three marking characterizing values are 
taken as features for the marking type classification. The needed information 
to determine the features are available in the lane border measurement points 
found in the course of lane recognition. The measurement points are first 
transformed to the lane (world) coordinate system before a heuristic method 
extracts all feature values of a lane border. That means we get for each mark- 
ing measurement point a marking line width value and for each marking line 
segment and gap a length value. The extraction of segment and gap lengths 
benefits from the calculation of little edge segments within the lane border 
measurement process so that a one pixel accuracy can be achieved. The vari- 
ances are calculated by transforming at each extraction position a one pixel 
difference from image coordinates to world coordinates. 

For the classifier only one feature vector with the three characterizing width 
and length values are needed. Thus all the extracted values have to be merged 
in some way. We decided to use a Kalman filter for each feature (marking 
width, segment and gap length). Using a Kalman filter has also the advantage 
to time average the features so that occasionally false lane detection meas- 
urements or partially rubbed off markers are smoothed out. 

Finally a nearest neighbour classifier determines the lane marking type. A 
nearest neighbour classifier calculates distances between the feature vector to 
be classified and sample feature vectors which represent different classes. 
Since we have variances of each feature a Mahalanobis distance function is 
employed. A Mahalanobis distance weight the distance according a covariance 
matrix (here the matrix is diagonal). That means high variances result in high 
distances. 

For each marking type class a set of sample feature vectors must be specified 
in a configuration file. E.g. the file entry 
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dashed_3to9m = { { 3, 9, 0 . 1} , {3, 9, 0.2}, {3, 9, 0.3}} 

specifies a marking type class with line segment length of 3 m and line seg- 
ment gap length of 9 m and marker width between 10 cm and 30 cm. 




Fig. 11. Marking type "dashed_3to9m": Marking segment length = 3 m 
and marking gap length = 9 m 



7 Object Mask Generation 

The function of this module is not to classify obstacles or other traffic partici- 
pants. Areas on the road are detected and marked in an object mask image 
where objects are located. The main purpose is to support the lane recognition. 
The lane recognition cannot decide whether a measurement point is a point on 
the lane border or e.g. a vertical edge of a car driving in front of the own vehi- 
cle. A wrong measurement point can cause estimation errors. To avoid these 
errors lane detection search areas (scan lines) are trimmed where the object 
mask image is marked. 

For the object mask generation an approach is chosen that analysis motion- 
compensated difference images assuming the vehicle moving on a planar road 
surface. The used motion compensation algorithm computes the required 
image warping parameters from an estimate of the relative motion between 
camera and ground plane. This is done for the last two consecutive images. 
The algorithm estimates the warping parameters from displacements at image 
corners and image edges. 
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8 Use of Digital Map Data 

Besides the use of map data for the lane keeping decision module, the lane 
detection can also benefit of it. All lane tracking systems have certain fixed 
configuration parameters. The determination of a configuration parameter is 
often a trade off between different scenarios concerning environmental and 
lighting conditions. A parameter value may work well in one scenario while 
the same value doesn't work at all in another. So the general idea to enhance 
the performance of lane detection systems with map data is to adapt certain 
configuration parameters with respect to information about the vehicle envi- 
ronment coming from a digital map. 

Several critical scenarios were identified in preliminary tests of state-of-art 
Lane Vision systems. Most of the problems were encountered in scenarios 
where external elements have visual features similar to lane markings or bor- 
ders like guardrails and sidewalks. In these scenarios the resulting lane mod- 
els deviate considerably from the real lane. If we would have some knowledge 
of the current lane in front of the vehicle, the estimated lane model could be 
validated. Here the digital map comes into play. Out of the GPS localisation and 
the digital map good estimates of the current lane width b and lane curvature 
c 0 can be provided. 

If map data and a GPS signal are available the c 0 and b tolerance intervals of 
the lane state plausibility check (chapter 5.5) and the Hough accumulator 
parameter limits for c 0 and b are dynamically adapted. If the lane detection 
now starts tracking other elements and the estimated lane width and curva- 
ture get out of range, the state plausibility check triggers a re-initialisation of 
the system. Because the Hough accumulator space is limited in c 0 and b coor- 
dinates to the same values as the tolerance intervals, the chances are good that 
a proper lane detection restarts. 



9 Conclusions 

We proposed a lane detection system aimed at being used in lane keeping 
assistant systems. The development focus lay on a high system adaptability 
with respect to the vehicle platform and different environment and traffic sit- 
uations. A clear separation of the camera calibration and vehicle position esti- 
mation is made to meet the vehicle platform flexibility requirement. The com- 
bination of a new initialisation algorithm and the use of positioning and map 
data lead to very robust detection results in varying traffic scenarios. The lane 
detection was furthermore extended to track the direct neighbour lanes as 
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well, an obstacle area detection algorithm was added and a lane marking type 
classification was introduced. One SAFELANE demonstrator vehicle (a truck at 
Volvo Technology in Sweden) is already equipped with our lane detection sys- 
tem and the so far conducted tests are very promising. In the next SAFELANE 
project phase extensive evaluation tests will be conducted on different vehicle 
platforms. 
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